Product Stabilization through Data Driven Testing

Provided services

Client

The European market leader in telecom management technology (Amsterdam, the Netherlands).

Product

Cost control and accounting system of telecom services. It is a group of apps: a web app for the end-users, a web app for admins, a server-side app for integration with other systems.

Business challenges

Product instability and the impossibility to quickly fix issues that appear in production negatively influenced the client’s reputation and the ability to increase sales.

Project technical characteristics

The entire app business logic was placed in the stored procedures (PL/SQL), which greatly complicated the debugging and maintenance costs of the code. The lack of Unit tests didn’t allow assessing the impact that changes in the stored procedures had on the entire app functionality. The CI/CD principles were not implemented, which also didn’t contribute to the Unit test creation. Complicated database architecture and lack of access to the production data didn’t allow developers to quickly reproduce bugs appearing there. All those points together led to the instability of the app and lots of challenges, which were difficult to find and solve. What was more, fixing one bug caused new bugs to appear.

Hard to reproduce bugs

Impossibility to fix the issues fast

Absence of Unit tests

Business logic located in stored procedures (PL/SQL)

DATA DRIVEN TESTING

1

Speeding up bug reproducibility We use production data without user information (GDPR)
2

Automation of the app life cycle CI/CD implementation (Jenkins)
3

Stabilization of the app We cover any code changes with DDT tests
4

Early bug detection, pre-production stage DDT test accumulation, system test coverage increase

Product stability control
Stable bug regression
Quick and efficient issue solving in production
Identification of bugs in the product at the development stage
Cost-cutting of test creation

Block diagram: Product Stabilization through Data Driven Testing

Approaches and solutions

Speed-up of bug reproducibility. Applying special utilities that delete user confidential data (we didn’t always have access to all the necessary data because of GDPR). As a result, we were able to use production data with full GDPR compliance to reproduce bugs.
Implementation of continuous integration and delivery (CI/CD) using the Jenkins server automation. It was the first step to systematic Unit test creation. With sufficient cover, Unit tests allow engineers to stabilize the app and conduct regression testing.
Data Driven Test coverage of any code changes. We ensured those tests helped us stabilize the product at the initial stage already, directing our efforts in the right direction: in case of safe and correct new bug fix, previous tests were successful. That way we increased control over app stability and made bug fixing fast and efficient. The usage of various database versions as reference values for DDT was another tech challenge, which made it possible to quickly add test data sets.
The last stage was the transition to the early error detection before changes were released to production. For that reason, we accumulated the number of DDT tests and test data sets, including separate tests for each bug. In doing so, we increased the system test coverage and ensured automation of regression tests.

Results and achievements

Test management was organized and test documentation along with a reporting system were created. It allowed addressing project challenges in a systematic and comprehensive manner.
Continuous integration and delivery (CI/CD) was implemented, which allowed for finding product bugs at the development stage.
Control over product stability was ensured. The implementation of Data Driven Testing stabilized the process of development and maintenance of the customer’s live-systems. Safe bug fixes allowed us to minimize the risks of new bugs in production, stabilize the product operation, and reduce the cost of test creation.
Stable bug regression was organized thanks to the systematic code coverage by Unit tests.
Data storage protection issues were considered. We applied special utilities to replace real data with fake test data so that the customer didn’t have to disclose any confidential user data to a third party. At the same time, we accelerated the process of finding errors in production.
Support and maintenance of the implemented approaches was transferred to the customer’s team as a part of the project delivery. We created the necessary documentation with tech manuals, diagrams, app work description (such as an installation manual, training videos and audios for quick immersion of new employees in the project). Later on, the customer’s team was able to independently develop and support automated tests.