Full-fledged Test Automation of a Product with Complex Architecture

Provided services

Client

IT product company (England, UK).

Product

Inventory and order management system for distributors and e-commerce, including ERP, CRM, and warehouse management system.

Business challenges

A complex scalable product with a large number of functionality and modules, not covered by automated tests. Constant release delays and bugs arising at the production stage negatively affected the customer’s reputation, as a result of which it was impossible to expand the client base and move to cooperation with larger companies (due to the higher error cost). Product development and delivery were intense and stressful for everyone involved.

Project technical characteristics

The system has a large codebase with a non-trivial, confusing architecture and mixed layers of code (business layer, database, controller, UI), which makes it difficult to maintain. Unit tests are almost absent, the Definition of Done didn’t have them, too. Requirements for the code level coverage are absent as well. Product quality control is carried out ad-hoc and with the help of manual testing. The system components are not isolated, due to the high level of cohesion the bugs appear at the slightest code changes. The product is very difficult to scale and integrate new functions to.

The principles of continuous integration and delivery (CI / CD) are not implemented, the product build and server updates are done manually, which negatively affects the speed of product delivery.

Constant release disruptions cause dissatisfaction among business owners and end clients; nervousness and stress arose in response to the pressure in the team. The situation is complexified by the lack of proactivity and technical leadership among engineers, and the accumulated technical debt over the years. Initiatives to introduce new engineering practices that benefit the project, in the long run, are ignored, leaving developers with learned helplessness.

Large codebase with non-trivial architecture and mixed code layers

Release delays and bugs at the production stage

Lack of test automation and CI/CD

Large amount of technical debt and learned helplessness in the team

1

Mediation of disagreements between customers to establish a culture of value-based product development Organizational, managerial and technical consulting
2

Optimization and reorganization of processes within the project team QA as the most important part of the product life cycle Building professional test management Improving the estimation process, introducing definition of done for the tasks, planning deliveries Transforming the business analysis process, introducing a unified standard for setting tasks
3

Automated build, regular QA and automation of product delivery CI/CD Implementation
4

Working on critical technical debts Creation of the framework for the application Unit testing Comprehensive coverage of all important system functionality with GUI autotests
5

Implementation of the best development practices and generation of a value-based culture Implementing Scrum methodology, organizing knowledge transfer within the team, encouraging the engineers’ initiatives

Regular and timely deliveries of new product functionality was established
Degradation of product quality at the production level was excluded
R&D to eliminate technical debts was initiated
A value-based approach to the development process in the team was established
Timely product development and scaling was ensured
Test automation process for providing quick detection of bugs when implementing new functionality was set up
The level of synchronization between the team and management was improved, as well as the psychological climate on the project

Block diagram: Full-fledged Test Automation of a Product with Complex Architecture

Approaches and solutions

We consulted stakeholders of the customer’s company on improving the current project situation. JazzTeam CTO acted as a mediator – an independent party, promoting an understanding among product owners with different points of view. Each of them recognized the need to improve processes, but the options they provided did not help eliminate the problem root. First and foremost, it was necessary to shift the focus from achieving short-term business goals to establishing a culture of value-based development.
We reorganized the processes:

Quality control was to become a critical part of the project lifecycle. Before JazzTeam joined the project, testing processes were carried out ad-hoc, the role of a tester was performed by developers and one of the product owners. We analyzed the product quality and instability related problems and offered customers to expand the team with manual and automated testing specialists. Our company took on the task of building high-quality test management.
To deal with the constant release delays, we decided to revise the task estimation process so that all team members could realistically understand the current status. A unified table of possible risks was introduced in addition to a clearer decomposition of labor costs into types of work (BA, design, development, automated and manual testing, bug fixing, etc.). For each risk of the task we set the probability for that risk to occur, which influenced the initial estimate. Also, each type of work got a list of criteria for completing the task (Definition of Done), which allowed for organizing systemic acceptance. All of those measures provided realistic estimates that enabled delivery planning without delays.
The business analysis process was qualitatively transformed. Instead of a schematic, concise, and incomplete problem statement, which was understandable only to employees with extensive experience on that project, we decided to introduce a unified statement standard, transparent and readable by the entire team. That allowed for better integration of the development and testing teams, provided clearer acceptance, significantly improved synchronization with the end customer for the desired result, and used the formulation to create test cases that were the basis for manual and automated testing. It also reduced the costs of product support and maintenance.

We implemented CI/CD for automatic build, regular quality control, and automation of product delivery. That process was complex, painful, and required a number of studies to achieve full build automation, including those on ensuring automatic build of product executable files and implementing LiquiBase for version control at the database level. It was necessary to overcome distrust on the part of the team, as well as insist on the mandatory use of CI/CD for everyone.
A large volume and lack of a culture of technical debt management made it critically difficult to work with it consistently throughout each sprint. At first, it was necessary to earn the trust of the team and, with small iterative changes, demonstrate the possibility, prove the importance of working through the debt, even in the conditions of constant developers’ lack of time with commercial tasks. We focused on the integration of automatic build of product executable files and were able to implement it at low cost, while the final effect allowed us to start applying CI/CD. It was appreciated by customers. Also, despite some resistance, engineers started to examine several researches for 1 hour a day. That approach allowed us not to lose focus on the commercial task delivery, and at the same time, we consistently moved forward to resolve the accumulated technical debts. Below are some illustrative examples of how critical tech debt was dealt with:

Creating a framework for app blackbox testing, which enabled us to start creating automated backend tests. Without that framework, developers could not create Unit-tests due to the large codebase, with a non-trivial, confusing architecture and mixed code layers.
Tests were run as a part of regular project builds (on schedule and on-demand) to ensure the stable operation of the product with new changes.
Full and regular GUI automated test coverage of all important system functionality. We created a framework to quickly write and maintain a large number of GUI automated tests with minimal costs. It helped increase the base of UI automated tests from zero to several hundred in a short period of time (several months). The framework was based on the PageObject pattern and consisted of a layer for accessing stored application data, a service layer that integrated the data layer and a layer of utilitarian methods to simplify management and further development of automated tests.

We implemented the best development practices and generated a culture of value. In agreement with the business owners, we decided to strengthen the role of engineers on the project and increase the degree of team self-organization. To do so, we applied the best approaches of the Scrum methodology. Daily stand-ups increased the synchronization level of the geographically distributed team. A demo of the new functionality in front of the entire team helped achieve a common understanding of the current project status. Misunderstandings and problems in the team were promptly resolved during regular retrospectives. Opinions of all project participants were heard and the product owners began to take into account and think over the suggestions and comments of the engineers. We established a systematic transfer of knowledge within the team – it comprised lectures and meetups on technological or domain expertise. All that helped improve the psychological climate and overcome the learned helplessness of engineers.

Results and achievements

Thanks to the carried out work, the customer was able to deliver new functionality regularly and on time, thereby developing and scaling the complex product. Quality degradation at the production level was eliminated. It opened up opportunities for ensuring the market share of the customer’s company and improving cooperation with larger players.
The value-based approach to development was established in the team. Shifting from short-term goals to strategic product development the product owners began to listen to the initiatives proposed by the team that delivered long-term value. Also, the product owners began to invest in the implementation of the necessary engineering practices (CI/CD, Unit testing, test automation). Eliminating technical debt was conducted during each sprint, and long-term technology researches were generated.
Thanks to the introduction of CI/CD, the process of the product build and delivery was automated. It became fast, transparent, and secure. That allowed ensuring the regularity of releases.
The complex app codebase was covered with Unit tests, thanks to which we stabilized the quality of the system when making changes.
Full-fledged test automation significantly improved the quality of the product and reduced the time for app development and support. Systemic test coverage made it possible to detect bugs quickly when implementing new functionality. That provided the opportunity for well-timed development to keep the product competitive.
We improved the synchronization level between distributed team members and product owners. Engineers became motivated to overcome learned helplessness, use a proactive working approach. The psychological climate in the team has been improved, the development process takes place in a more relaxed atmosphere without unnecessary stress and nervousness.