Telecommunications Data Processing Center

Provided services

Backend Development, Test Automation Services, Manual Testing Services, Project Management, CI/CD Implementation & Modernization. DevOps Services

Client

A cybersecurity and network traffic analytics company from the United States. The company uses cutting-edge technologies to monitor network activity, analyze user behavior, and prevent malicious activities. Its customers are state and private organizations looking to proactively protect themselves from potential threats using the company’s cybersecurity and analytics products.

Product

The client was working on a multi-module, multi-version, and cross-browser network traffic monitoring solution. It was deployed to various business environments to maintain security and data privacy while shielding against potential cyberattacks.

The product is very popular among large telecom companies and is supplied to them to ensure integral network and data protection.

Challenge

JazzTeam was invited to the project to automate the product’s deployment. Our main objectives were streamlining development, CI/CD processes, and quality assurance.

The product was delivered to a variety of businesses each with different hardware configurations, infrastructure, and network topology. The original delivery process carried out by our client was partly manual and very complex. So it was often the cause of delivery time delays and software malfunctioning.

One of the most challenging business cases involved deployment to at least 11 different servers bound by an intricate network configuration.

Automating delivery across these diverse environments can be quite challenging. For example, setting up the app for Linux and SunOS platforms has nothing in common whatsoever. And each server had its own characteristics.

The client was struggling because of the countless number of settings, and it was hard to immediately determine what was causing the bugs (Is it the environment? Application configuration? Functionality bugs? Something else entirely?).

In the end, one person had to be involved in managing servers and passing automated tests. Because of the complexity, the client eventually switched to fully manual testing which drastically slowed down delivery to customers. Integration was done manually as well.

Our team was dispatched to integrate the best CI/CD practices and streamline the automatic deploy and delivery pipeline from front to back.

Solution

JazzTeam’s experts became an integral part of the client’s team. We oversaw the product’s development and deployment processes and made sure new versions were regularly delivered to the client’s customers.

To succeed, our experts analyzed the project’s documentation, audited current processes and pipelines, and created a roadmap for optimizing development, testing, and deployment.

The highlight of our collaboration was creating a CI/CD configurator, which allowed us to wrap the most complex configurations into a simple interface that the admins could use to save time and easily create the setups needed. This also diminished the amount of mistakes made and critically improved the stability of the product.

Our DevOps engineers and developers carried out a range of initiatives across several phases:

Phase 1. Improving the project’s CI processes

Our team introduced methods and tools for complete product build process automation. The environment was configured from a standard template where the app was installed automatically. There was practically no more room for human error.

Because of our efforts, server configurations became version-controlled, documented, and repeatable. New product versions could now be added with just a few lines of code using a parameterized timer.

The automated infrastructure as code pipeline allowed us to consistently set up new environments using the same playbooks and easily manage their configurations separately via the popular and efficient Git Branches approach.

Stages of CI/CD

The customer’s infrastructure was monitored using a multi-component system that we configured from scratch. We used various metrics from remote hosts to proactively inform us of system operation issues. The system indicated exactly which service was causing the issues, allowing the engineers to quickly remediate any faults in operation.

Our new CI framework simplified the environment configuration and minimized the risk of environment-related errors. It also saved massive amounts of time on product builds, which allowed engineers to focus on more important tasks.

This CI process modification allowed our clients to gain a 10X accelerated delivery in a variety of business cases.

Phase 2. Introducing a Configuration Builder with seamless UI for selecting deployment options

JazzTeam created a special app with a convenient user interface displaying all the possible options and settings for deployment. The settings were selected from drop-down lists or entered manually.

Based on the rules selected in the app’s menu, the application generates a configuration file used further in the CI/CD pipeline.

Accurate configuration and visualization of the deployment process for IT products with complex delivery organization

With the app in place, we made it possible for virtually any engineer of any level to easily configure the required environment settings. It decisively decreased the amount of errors encountered during manual configurations.

Most of the manual behind-the-scenes configurations were automated, so setting up for various network topologies became very simple.

Phase 3. Manual tests coverage

Not everything could be covered with automated tests. Especially not at the start of the project. So our team was responsible for creating test coverage based on the customer’s requirements and the product’s documentation.

We had to make an overhaul of the whole testing framework as it was inefficient:

Tests covered only a part of the app’s functionality;
Testing was carried out irregularly;
The test success rate was as low as 10%;
Most of the tests were inconclusive (no indication of cause);
It took too much time to figure out the cause of errors;
Automated tests were run manually for each server setup.

To improve performance, we also needed to manage a greater number of test cases. Our team solved this with the transition to more efficient test management tools (we moved tests from TestLink to Zephyr).

The tests were performed using virtual machines, so the process avoided the extra hassle of configuring physical servers.

Later on in development, the manual test cases we created were used in test automation.

Phase 4. Implementing automated tests

Since product delivery was already partially covered with automated tests by the client, we set out to restore and optimize these tests to fit them into the new test automation pipeline.

JazzTeam QA professionals were tasked with refactoring the automated test code, expanding the logging system, and improving exception processing. Based on the manual test cases set up before, we created automated tests that covered all the important functionality running nightly for each new build.

Our team also containerized the autotests in Docker to achieve faster test execution, more efficient resource consumption, and reduced maintenance overhead compared to the traditional virtual machine approach used by our client. The automated tests were created in Selenium and CLI.

The new system delivered consistent test environments and provided a much more reliable quality assurance process.

You can find a detailed overview of how automated testing of this product was performed in our dedicated case study.

Technological Overview

To set up the Infrastructure as Code approach, our team implemented Ansible in conjunction with Git. We also replaced the Jenkins UI configurable pipelines with Groovy implementation.

We used Python for the back-end and jQuery for the front-end of the Configuration Builder. It generated YAML files that specified which version should be deployed on which server and with what conditions. These files were fed to another component based on Ansible scripts. This way, we synchronized configurations within the pipeline.

To containerize automated tests, we gradually moved them from the VM environment to Docker. Our team devised a special solution that decomposes the existing VM infrastructure of autotests into a relational database. This way, the system could retrieve the decomposed data, serve the autotests directly to Docker, and instantly run them.

The full cycle of launching test plans looked like this:

Running Docker containers (the system performed load-balancing of available resources);
Generating individual test execution tasks;
Sending and receiving results through Jenkins;
Disabling containers and saving test results back to the database.

Infrastructure monitoring was set up by us from scratch using Nagios. Metrics were collected using active checks via the SNMP protocol.

To eliminate the hassle of regularly removing outdated nightly builds, we set up the storage for nightly snapshots in the Nexus repository. Outdated snapshots were automatically removed from the repository while suitable builds were manually upgraded to release status.

Our team implemented test builds using CMake. It allowed us to make builds and send reports to the developers at almost every change in the repository. TestNG was used to implement test grouping and reduce dependencies between different test results while also eliminating code duplication.

Result

Automated CI/CD processes allowed our client to build and test the app without depending on manual setups and do it much quicker. Several versions could be deployed simultaneously. Many cases showed a 10X accelerated delivery.

We configured the solution so builds would be fully automated and happened once a day on schedule. Plus developers could trigger builds whenever they needed and get timely feedback on their code commits instead of waiting around for reports from QA.

We automated pretty much every step of getting the product to users, which was a game-changer — bottlenecks like depending on specific team members to handle deployments and human errors were eradicated.

Various report types were automatically sent to engineers and management. All members of the distributed team had up-to-date information about the product’s performance and stability.

We are proud to share our achievements on this project:

More than 30 test environments (>100 virtual machines) were configured for performing manual and automated tests.
All 6 application deployment options were automated using our Configuration Builder.
The time needed to fix discovered bugs has been cut down by over 50%.
The average number of bugs being found decreased by 30%.

Seeing the great results on this project, the client offered our team to continue the collaboration further and fine-tune the CI/CD and test automation for another product.