IT product company, USA (California).
Software for Internet traffic processing.
Test automation implemented earlier by the customer’s team didn’t help stabilize the operation of a dynamically changing, multi-component web app and didn’t give full confidence in new versions delivery quality, which led to a decrease in product release frequency.
Project technical characteristics
Test automation was implemented by the customer’s team, but the potential and value of the implemented automated tests were not fully used. The created automated tests covered only a part of the basic app functionality and were launched irregularly, only 10% of them were successful. The full status of test passing was unclear: it wasn’t possible to establish right away which test failure indicated the problems in the app, and which ones indicated issues with the environment. It took too much time to identify the reasons for the failed tests. Automated tests were run manually by each specialist on separate servers with different configurations.
The product build took a lot of time and was centered around one person who had to perform a lot of manual actions. Managing servers and passing automated tests was a long and labor-consuming process, so bugs in the app were mostly identified through manual testing, which greatly slowed down the product quality verification.
Approaches and solutions
- Restoration and optimization of the current test automation system:
- Search and analysis of the reasons for bug appearance.
- Refactoring the automated test code and expanding their logging system, improving exception processing.
- Test division into groups to reduce dependencies between the results of their passing and eliminate code duplication.
All that allowed for ensuring the operability of automated tests, the ability to quickly change them and keep them up to date.
- Extending the infrastructure for automated build, testing, and delivery (CI/CD) of application versions using Jenkins:
- Complete product build processes automation, environment creation, and component installation for automatic app build.
- The build of the product was carried out by trigger after the commit and on a daily schedule with the email notification of the development team about the results, which saved time on product development and eliminated the influence of the human factor on the build process.
- The server was created from a standard template, the product was copied and installed on the servers automatically, which simplified the environment configuration and minimized the risk of environment-related errors and problems.
- Testing process improvement, necessary documentation update and composition. We initiated the transition to a more efficient test management tool (from TestLink to Zephyr) to monitor a larger number of test cases. This allowed us to easily and quickly analyze the reasons for the test failure. Also, instructions and training materials were created to quickly immerse new specialists in the project.
- Automated tests coverage of all the important functionality (both GUI and REST API) to minimize the risk of critical bugs.
Results and achievements
- The introduction and improvement of automation processes for product build, testing, and delivering allowed us to establish the regularity of releases and increase their speed.
- The work and support of automated tests created by the customer’s team were restored. All the important functionality was covered with automated tests, automatic product quality check was carried out daily, which allowed us to quickly find and fix bugs upon the occurrence and ensured confidence in the quality of the delivered versions.
- Continuous integration and delivery (CI/CD), was implemented, an environment for automating the app build and testing was installed and configured. The maximum number of steps when delivering a product to the end-user was automated, the influence of the human factor and dependence on one team member was excluded. Automatic build was carried out daily on a schedule and on-demand, which significantly saved time for product development (developers immediately received feedback on the commit result).
- The testing process was built, the test documentation was improved (the reports became more informative and structured). Various report types were automatically sent to engineers and management, all members of the distributed team had up-to-date information about the product state and stability.