Telecommunications Data Processing Center

Project summary

thumbnail-image The customer is developing a complex multi-module system. This solution is supplied to large telecom companies and is highly popular. Because of the domain area specifics, the product has a non-trivial delivery scheme and is installed on a minimum of 11 machines in the most complex topology.

The customer needs to organize the automatic deployment of the system on client company servers. Moreover, it is necessary to consider a variety of features: each client company has different hardware configurations for storing large amounts of data. In addition, it is necessary to ensure the proper work of the application through manual and automated testing.

JazzTeam specialists became a part of the customer’s team and are engaged in the following activities on this project:

CI process improvement. The customer implemented continuous product integration, but the team continued doing many of the activities manually. So, it was necessary to optimize this process too.
Development of an additional application (Configuration Builder) for visualizing and configuring the deployment process.
Comprehensive and regular manual testing of the application.
Creation and regular execution of automated combined tests for the web interface and Command Line Interface (CLI).

Product features

Multi-module, multi-version, and cross-browser compatibility. The application has 6 deployment options. For example, one of the options means the deployment of different components of the application on 11 machines combined in a cluster.
The product compilation and building may take several hours.
Using a special distribution package of the operating system with the focus on minimalism, fail-safety, and security.
The main function of the application is working with network traffic. A special tool which generated network traffic was used for the automated tests.

Accurate configuration and visualization of the deployment process for IT products with complex delivery organization

Project features

Supporting CI cycle for Linux and SunOS platforms. The build and deployment for each OS had nothing in common.
Using Jenkins CI and lots of plugins for it.
Specific application area of the product. To participate in the project, our specialists had to immerse themselves in it and have expert knowledge of the specifications, every function, and networking standards of the system.
One of the main challenges of the project was troubleshooting. Because of the large number of settings, it was not immediately possible to determine what was causing the bugs (either an environment problem, an application configuration error, bugs in applications, or something else…).
Administration of Unix-like systems.
A distributed development team, whose members were located in different time zones with a maximum time difference of 11 hours.

Approaches and solutions

1. Improvement of CI process on the project

1.1. Implementation of Ansible. For continuous deployment and configuration of the product, our team implemented Ansible in conjunction with Git (especially with the mechanism of branches). Today, the JazzTeam team successfully maintains several playbooks versions, each with its own features.

1.2. Implementation of the Infrastructure as Code concept. We replaced the Jenkins UI configurable pipelines with Groovy implementation. This resulted in the following benefits:

The pipeline code is found and managed through Git.
No code duplication.
New product versions are added with a few lines in the parameterized timer settings.

1.3. Transfer of automated tests to the Docker Tests system. The customer is gradually transferring automated tests from the VM environment to Docker. We have developed a special system which decomposes the existing VM infrastructure of autotests into a relational database. Based on the decomposed data and according to the settings entered by the operator through the UI, autotests are run already in Docker. The main functions of the system:

Decomposition of the existing infrastructure of autotests into the database.
Test plans setting through the UI.
The full cycle of launching test plans:

running Docker containers (the system performs load-balancing of available resources);
generating individual test execution tasks;
sending and receiving results through Jenkins;
disabling containers, saving test results to the database.

Stages of CI/CD

1.4. Monitoring with Nagios. The customer’s infrastructure was monitored using a multi-component system that we configured from scratch based on Nagios. Various metrics from remote hosts are collected using active checks through SNMP protocol.

1.5. Storage of artifacts in Nexus. Together with the customer, we set the saving of nightly build artifacts (snapshots) in the Nexus repository. Nightly builds deemed suitable for release are manually upgraded to the (releases) status. Outdated snapshots are automatically removed from the repository. This solved the problem of the need to regularly manually remove old nightly builds, which were not included in the release.

1.6. Test builds based on CMake. Together with the customer, we are implementing a new build system based on CMake, which will significantly reduce the build time. It allows us to make builds and inform the developers about the result after almost every change in the repository.

2. Development of the Configuration Builder application for visualizing and configuring the deployment process

Configuration Builder is an application that creates a configuration file and automatically (using scripts) executes certain commands on various servers. The system components were written in Python, the GUI was created in jQuery.

Our team created an application with a special menu, which conveniently displays all the possible options and settings for deployment. The settings are selected from drop-down lists or entered manually. Based on the rules selected in the menu, the application generates a configuration yaml file. It specifies which version should be deployed on which server and with what conditions. This yaml file is read by another application component based on Ansible scripts. Thus, the yaml file is synchronized with the pipeline. Further, the rules originally specified by the user will be automatically considered in the customer’s CI/CD process, which will significantly simplify the delivery of new product versions.

3. Manual testing of the product

Our team is responsible for creating test coverage based on the customer’s requirements and official documentation. Later, the cases created by the manual testing team are used for test automation. The manual testing process has the following features:

Applying the Man-Machine Language used to run the product (creating and deleting entities).
Flexible work with all product versions on physical and virtual machines (Linux and SunOS systems).
All testing work was done on remote virtual machines without the need for complex configuration of the environment.
Our team initiated a transition to a more efficient test management tool (from TestLink to Zephyr) for managing a big number of test cases.

4. Creation of automated tests

Based on the test cases created as part of the manual testing process, we wrote the combined Selenium + CLI automated tests, which are run nightly for each new build. You can find the details of how automated testing of this product was performed in our case study.

Project results

The Continuous Integration process was set up and implemented. The customer team was trained and documents on the use and administration of the CI process on the project were created.
A big number of test cases were created and are regularly used for the project’s each new functionality.
A manual testing process was organized, which allowed us to identify bugs on a regular basis and promptly fix them before releases.
More than 30 test environments (>100 virtual machines) were configured for performing manual and automated tests.
The universal solutions within Jenkins CI were implemented, which minimizes manual activities for CI administration and application deployment.
All 6 application deployment options were automated using our team’s solution for configuring the deployment process.
The application building process was improved. It became possible to build several versions simultaneously. Also, the building of patches for old versions of the application was fully automated.

Company’s achievements during the project

During the project implementation, we proposed and implemented many improvements, which made it possible to optimize development and testing on the project.
JazzTeam specialists learned a lot of new technologies.
While solving the main tasks, the JazzTeam team created a universal solution for configuring and visualizing the deployment process.
As a result of successful completion of the project tasks, the customer offered our team to continue the cooperation and implement continuous integration and test automation for another product.