70% Costs Cut on Bug Fix and Support through CI/CD Implementation

The project represents a system designed to simulate the behavior of a physical object under specific conditions. The project had already been functioning at the time JazzTeam joined it to increase the product stability and improve development processes.

Instability was one of the most important and difficult problems. Deliveries were irregular, it was difficult to predict the application behavior when the code was modified. Therefore, first of all, CTO of JazzTeam took a decision to implement automated testing within the CI/CD system.

The main reasons for introducing automated testing were as follows:

Each instance of the application is unique, has its unique serial number and USB hardware key, so it was an important factor affecting the entire CI/CD system. This imposes the following restrictions:

As part of Continuous Integration the main objective was to implement daily automated quality control via Unit and UI testing. It was needed to achieve the following important strategic goals:

As part of Continuous Delivery the main goals were as follows:

To implement the CI/CD practice, it was also required to set up management processes. Work process organization – from tasks setting to testing, implementing Scrum, changing the team’s mindset.

Before JazzTeam joined the project, it was a normal practice to do everything quickly, without documentation and elaborated testing. Naturally, much time was spent for the support due to regular problems emerging during the delivery. Three months later the development team began to reduce the technical debt and maintain the necessary documentation, while the speed of creating new functionality remained the same. As a result, the number of delivery problems significantly reduced.

A mandatory step in the development of new functionality was its coverage with Unit tests and UI automated tests.

Key achievements on the project during the first six months were:

PRODUCT CHALLENGES

Difficult to scale and develop the product due to the large number of bugs

Quality control processes were not set up, there was no CI/CD

Lack of Test Automation (UI/Unit tests)

Constant release delays in the process of product delivery

SOLUTION STAGES

• 1
  Establishing a value-based approach to the development process Organizational, managerial and technical consulting
• 2
  Setting up the processes Engaging our company's manager to manage the project, implementation of Scrum and test management optimization
• 3
  Changing the team's mindset Self-organization of the team, implementation of the best engineering practices and approaches, overcoming learned helplessness, constant work with technical debt
• 4
  Automation of application development lifecycle CI/CD Implementation
• 5
  Quality assurance for the product Implementation of UI/Unit testing practice by developers using the DDT approach

RESULTS

• 
  Barriers to regular building up of the product's functionality were eliminated
• 
  QA processes were organized, Scrum and CI/CD were implemented
• 
  Release speed was increased by 5 times
• 
  Stable functioning of the product, the customer was confident in the quality of each delivery
• 
  A value-based approach to the development process was formed, the psychological climate in the team was improved

Block diagram - Consulting on a Science-intensive Product Development and Building a QA Process

Well managed testing allowed changing the team’s view of the development process. Now, when fixing a bug, it is an element of good style to completely cover this part of the code with tests in order to eliminate the risk of bug reoccurrence. Also, as mentioned above, when seriously interfering in an already working component, it became best practice to cover it with tests first and only after that introduce changes in it.

Since the system works with scientific data, an active use of Data Driven tests became an important area of testing. So, we were able to involve the entire project team in the testing process.

After each sprint, metrics on changes in code coverage by tests are recorded, so the code coverage trend is tracked. We use the following two key parameters as reference metrics: code lines coverage and condition coverage.

Progress is not fast, as the project has a very large legacy code base – several hundred thousand lines of code. Therefore, coverage increased by only 2 percent in two months. However, the number of Unit tests increased by 25% in absolute figures. Surely, these are not the most impressive figures, but our main goal was to change the team’s approach to writing Unit tests and integration DDT tests. And we achieved this goal!

The process of UI tests implementation started with a practical investigation conducted by the team. The essence of the study was to choose two UI testing frameworks and write a few UI tests using each of them. Then it was required to compare the results and choose the only one of the frameworks. For this project we decided between free Appium and White frameworks. Upon practical tests we chose White framework.

The process was organized as follows: the developer allocated 5 hours a week for the tasks related to writing UI tests. The results were tracked every day at the stand-up and entered into a special table, which was printed out and placed near the Kanban board.

At the first stage only Smoke UI tests (basic, most important) were written, which, however, allowed us to check the performance of the finished application instance for the customer in automatic mode, and not manually as before.

At the second stage we implemented writing UI tests as one of the criteria intended to check feature readiness for testing. As far as new features were developed, it allowed to gradually cover the entire functionality of the application with UI tests.

A completely new practice for the project was to create a test instance of the application with hardware protection enabled. At the same time a test USB security key is flashed. After that the test application is installed on a separate machine and then the fully ready-made application is tested in the operating environment using UI tests. This made it possible to identify possible defects at an earlier stage. It also made the development process faster, more transparent and predictable.

Before that, to test the application with an already working protection system and all the components, the developer had to create it first. It required much effort and was performed only at the end of the development process, before submitting the application for testing. Now the developer can run UI tests on the ready-made application at any time. Taking into account the minimum labor intensity, this became an element of good style. Also, UI testing became mandatory before merging into the main development branch.

The introduction of Continuous Delivery greatly simplified the process of distributives creation. Any user, not just the developer, can create a new instance of the application.

In addition, automated testing of new copies of the application significantly increased the productivity of the team and made it possible to release twice as often.

Stack: .NET, C#, C++, Managed C++.

Infrastructure: Windows, Jenkins.

Test Automation libraries: Appium, White.

After CI/CD implementation we achieved the following results: