Test Automation Optimization accompanied by CI/CD Integration

Provided services

Client

IT product company, USA (California).

Product

Software for Internet traffic processing.

Business challenges

Test automation implemented earlier by the customer’s team didn’t help stabilize the operation of a dynamically changing, multi-component web app and didn’t give full confidence in new versions delivery quality, which led to a decrease in product release frequency.

Project technical characteristics

Test automation was implemented by the customer’s team, but the potential and value of the implemented automated tests were not fully used. The created automated tests covered only a part of the basic app functionality and were launched irregularly, only 10% of them were successful. The full status of test passing was unclear: it wasn’t possible to establish right away which test failure indicated the problems in the app, and which ones indicated issues with the environment. It took too much time to identify the reasons for the failed tests. Automated tests were run manually by each specialist on separate servers with different configurations.

The product build took a lot of time and was centered around one person who had to perform a lot of manual actions. Managing servers and passing automated tests was a long and labor-consuming process, so bugs in the app were mostly identified through manual testing, which greatly slowed down the product quality verification.

Test automation process previously implemented by the customer was inefficient

Manual and irregular run of autotests on separate servers with different configurations

Product build took a lot of time and was tied to one person

Low speed of product quality check, rare releases

1

Restoration and improvement of the current test automation system Search and analysis of bugs reasons, refactoring of the autotests code
2

Block item Optimization of automated building, testing and delivering processes using Jenkins
3

Improvement of the testing process Writing the necessary documentation, implementing efficient test management tools
4

Prompt detection and localization of critical bugs Autotest coverage of all important functionality (both GUI and REST API)

Regularity of product releases and acceleration of their delivery
Product stability and confidence in the quality of the delivered versions
The product build was accelerated, the influence of the human factor on the build and delivery process was excluded
The cost of the development process was reduced
The testing process was built, the test documentation was improved

Block diagram: Test Automation Optimization Accompanied by CI/CD Integration for the Product with Sophisticated Build and Test Approaches

Approaches and solutions

Restoration and optimization of the current test automation system:

Search and analysis of the reasons for bug appearance.
Refactoring the automated test code and expanding their logging system, improving exception processing.
Test division into groups to reduce dependencies between the results of their passing and eliminate code duplication.
All that allowed for ensuring the operability of automated tests, the ability to quickly change them and keep them up to date.

Extending the infrastructure for automated build, testing, and delivery (CI/CD) of application versions using Jenkins:

Complete product build processes automation, environment creation, and component installation for automatic app build.
The build of the product was carried out by trigger after the commit and on a daily schedule with the email notification of the development team about the results, which saved time on product development and eliminated the influence of the human factor on the build process.
The server was created from a standard template, the product was copied and installed on the servers automatically, which simplified the environment configuration and minimized the risk of environment-related errors and problems.

Testing process improvement, necessary documentation update and composition. We initiated the transition to a more efficient test management tool (from TestLink to Zephyr) to monitor a larger number of test cases. This allowed us to easily and quickly analyze the reasons for the test failure. Also, instructions and training materials were created to quickly immerse new specialists in the project.
Automated tests coverage of all the important functionality (both GUI and REST API) to minimize the risk of critical bugs.

Results and achievements

The introduction and improvement of automation processes for product build, testing, and delivering allowed us to establish the regularity of releases and increase their speed.
The work and support of automated tests created by the customer’s team were restored. All the important functionality was covered with automated tests, automatic product quality check was carried out daily, which allowed us to quickly find and fix bugs upon the occurrence and ensured confidence in the quality of the delivered versions.
Continuous integration and delivery (CI/CD), was implemented, an environment for automating the app build and testing was installed and configured. The maximum number of steps when delivering a product to the end-user was automated, the influence of the human factor and dependence on one team member was excluded. Automatic build was carried out daily on a schedule and on-demand, which significantly saved time for product development (developers immediately received feedback on the commit result).
The testing process was built, the test documentation was improved (the reports became more informative and structured). Various report types were automatically sent to engineers and management, all members of the distributed team had up-to-date information about the product state and stability.