Organization of automatic deployment of the most complex system in the field of telecommunications

Client

A software product company, USA (California).

Products

Complex multi-module system in the field of telecommunications.

Business challenge

The customer is engaged in the development and delivery of an IT solution for large telecom companies. Due to the specifics of the domain area, the product has a non-trivial delivery scheme, for example, in the most complex topology, it is deployed on at least 11 different servers.

The customer needs to organize automatic deployment of the system on the servers of customer companies. At the same time, various features should be taken into account: each customer company has different configurations of equipment for storing a large amount of data. In addition, it is necessary to ensure high-quality operation of the application using automated testing.

Project features

A sophisticated product that includes an emulator of various devices at the TCP IP stack level.
The application has 6 deployment options. For example, one option involves deploying different application components on 11 machines that are united into a cluster.
The process of compiling and building a product can take several hours.
Using a special distribution of the operating system with a focus on minimalism, fault tolerance, and security.
The main function of the application is to work with the network traffic. A special tool was used for automatic tests that generated traffic on the network.
In the process of implementing the task, our team gained experience in implementing CI/CD in one of the most technologically challenging solutions we have encountered in our practice. It was necessary to develop our own utilities for implementing CI/CD, as well as a separate web application for high-level management of deployment configurations.

Organization of automatic deployment of a complex multi-module system on the servers of the customer's company, and ensuring high-quality operation of the application using autotesting

The most sophisticated product

6 deployment options

The main function of the application is to work with the network traffic

The process of compiling and building a product can take several hours

Creation automated tests (Selenium + CLI autotests)

1

Improvement of the CI process on the project Implementation of Ansible in conjunction with Git Implementation of the Infrastructure as Code concept Transfer of autotests to Docker Test system to save the company's resources Customer's infrastructure monitoring with Nagios Saving nightly snapshots in the Nexus repository Implementation of a new build system based on CMake, which reduces the time needed to build software by several times
2

Creating a Configuration Builder application to visualize and configure the deployment process Creating an application with a special menu, which conveniently presents all possible options and settings for deployment

The continuous integration process was implemented
30+ test environments (>100 virtual machines) were configured for testing
Universal solutions within Cl Jenkins were successfully implemented, which minimize manual activities for Cl administration and application deployment
All 6 options for application deployment became automated, and the possibility to collect several versions of the application at the same time was implemented

Block diagram: Organization of automatic deployment of the most complex system in the field of telecommunications

Approaches and solutions

Improvement of the CI process on the project. In order to continuously deploy and configure the product, our team carried out the following activities:

Introduced Ansible in conjunction with Git.
The concept of Infrastructure as Code was implemented. We replaced UI Jenkins configurable pipelines with Groovy implementation. As a result, the pipeline code is located and managed via Git, and new versions of the product are added with the help of several lines to the settings of the parameterized timer.
Autotests were transferred to the Docker Tests system. To save the company’s resources, our team transferred automated tests from the VM environment to Docker. For this purpose, our specialists developed a special system that collects information about the existing autotest infrastructure for launching in the VM, and saves the results of the analysis in a relational database. Based on the decomposed data, and in accordance with the settings entered by the operator via the UI, the system runs autotests already in Docker.
Monitoring with Nagios. The customer’s infrastructure was monitored using a multi-component system set up by us from scratch on the basis of Nagios. Various metrics from remote hosts are collected using active checks via the SNMP protocol.
Storage of artifacts in Nexus. Storage of nightly snapshots in the Nexus repository was set up jointly with the customer. Nightly builds recognized as suitable for release are manually upgraded to the status (releases). Outdated snapshots are automatically removed from the repository. So, the problem related to the need to regularly remove outdated nightly builds that were not included in the release was solved.
Check builds based on CMake. A new build system based on CMake was introduced jointly with the customer. It reduces the build time by several times. It can be used to build and inform developers about the result after almost every change in the repository.

Creating a Configuration Builder application to visualize and configure the deployment process. Our team created an application with a special menu, which conveniently presents all possible options and settings for deployment. The settings are selected from the drop-down lists or set manually. Based on the rules selected in the menu, the application generates a configuration yaml file. It specifies the version, server, and conditions for deployment. This yaml file is read by another application component based on Ansible scripts. So, the yaml file is synchronized with the pipeline. Then, the rules initially specified by the user will be automatically taken into account in the customer’s CI/CD process, which will significantly simplify the delivery of new versions of the product.
Creating automated tests. JazzTeam specialists prepared combined Selenium + CLI autotests, which are run for each new build every night.

Project results

The continuous integration process was configured and implemented. The customer’s team was trained, and documents on the use and administration of the CI process on the project were created.
30+ test environments (>100 virtual machines) were configured for testing.
Universal solutions within CI Jenkins were implemented. They minimize manual activities for CI administration and application deployment.
All 6 deployment options became automated with the help of the deployment process configuration solution created by our team.
The application build process was improved. The possibility to collect several versions at the same time was implemented. Also, the patch build for the earlier versions of the application became fully automated.