A leading provider of network services to digital infrastructure and telecommunications operators.
The “Application for Managing Products and Services” (hereinafter referred to as the Application) is part of a group of applications that support the business processes of telecom companies. These processes include enabling/disabling services, informing customers, and charging subscribers for mobile operators.
The Application consists of 2 services:
The Status Manager service receives information from adjacent services regarding changes related to subscribers. It then makes decisions about enabling or disabling certain product or service packages.
For example, a subscriber has activated a tariff plan that provides the basic services package (calls, SMS) even in case of a high debt. When the subscriber’s balance exceeds the allowable debt limit, the services included in the current tariff must be disabled until the debt is paid. In this scenario, the Status Manager is notified of the limit exceedance and sends a request to the adjacent subsystem to enable the basic service package for this subscriber.
- The Loader service performs a “cold start” of the Status Manager. It is necessary to load real data into the cache so that the Status Manager can start working with other third-party services. That is what Loader does. It downloads subscribers, their current status, and activated services with tariff plans from databases of other services in the system, and uploads them to the Application cache.
Given the complexity of the Application, deploying it to production without a stable automation framework would be extremely difficult. So, the client decided to proceed with the automation framework development.
To successfully develop and maintain such a framework, it was important to carefully select specialists with extensive hands-on experience in such projects. Seeking such specialists, the client approached JazzTeam.
We needed to perform the full cycle of test framework development, from requirements analysis to test creation. The requirements for testing were as follows:
- All the services must be run in Docker within one network.
- To run the services and Groovy scripts, it was necessary to use YAML files, which were supplied to the customer during the release.
- Third-party services of related teams must be closed by a Mock server.
For automated testing, it was necessary to develop a comprehensive test framework that would include all the required functions for creating automated tests. This test framework was required to be compatible with all the services integrated with the “Service for Managing Products and Services”, including Couchbase, Kafka, Oracle DB, ZooKeeper, Mock Server, and REST.
The task was complicated by the lack of a clear understanding of how the final product should function because the development was still in progress when the test automation work began.
In collaboration with the teams responsible for implementing other services, it was necessary to conduct thorough testing on the stands. The configuration of these stands closely resembled the customer’s environment.
Part of our solution was comprehensive testing, which included the following:
- Developing comprehensive test scenarios. The description of the scenarios was made in TestRail. In total, around a dozen scenarios have been described.
- Consulting other teams about the work and configuration of the Status Manager.
- Executing test scenarios and assigning bugs to developers from all the teams when these bugs were found during testing.
The test project was implemented using JUnit5 and SpringBootTest. The TectContainers library was utilized to run services in Docker. During the build process, before starting the tests, the parsing of YAML files was implemented. This resulted in the creation of all the required Groovy scripts in Zookeeper. Additionally, a configuration file containing the settings for connecting to third-party services was added to the Application container.
Here is a review of the test framework architecture:
- The configuration layer contains all Spring configurations responsible for running the testing infrastructure.
- The constants layer consists of classes that contain constant values used in the tests.
- The object_builders layer contains builder classes for test data. For example, it includes a builder for creating a JSON object that is sent to the Application during a request.
- The services layer contains classes for working with services such as different DAOs and clients.
- The utils layer contains Util classes.
- The YAML layer contains classes for working with YAML files.
The diagram below shows a description of how the Application works:
Description of the Status Manager operation
One distinctive feature of the Status Manager architecture is its flexible customization. Customers can independently configure the rules for enabling/disabling products and services, without needing to modify the core code of the service itself.
This is achieved by storing the rules description in Groovy scripts in Zookeeper. When a new script with rules is added, the Status Manager automatically uploads it into its memory.
Information about subscribers, enabled products, and their statuses is stored in the Couchbase NoSQL database.
When the Status Manager receives a subscriber change event through the asynchronous channel (RabbitMQ), it sends a REST request to enable a product for the subscriber. This is done based on the existing rules and the presence of the subscriber and their services in Couchbase. Afterward, the external service responsible for enabling products for the subscriber sends a request to the Status Manager. Based on this request, the enabled/disabled product is saved in Couchbase.
Description of the Loader operation
The Loader also offers a feature that enables flexible customization without modifying the core code. This customization is achieved by executing rules for downloading subscribers and services from external sources. These rules are described in Groovy scripts and stored in ZooKeeper. When a new script with rules is added, the Loader automatically saves it to its memory.
To start downloading subscribers and services, the user sends a REST request to the Loader. Depending on the existing rules, the Loader performs one of two possible operations:
- Downloads specific subscribers from the external Oracle DB and saves them to its own Couchbase DB.
- Performs a REST request to download the subscriber’s products from the external Products Downloading Service. Once the request is successfully executed, the Product Downloading Service begins sending the product data to Kafka. The Loader then saves this data to the Couchbase DB.
The test framework was meticulously developed, configured, and deployed, overcoming the inherent complexities of the application. To ensure comprehensive test coverage, test scenarios were created for the existing functionality, even though the development was still in progress at the start of the test automation work. We also conducted extensive research and development (R&D) to enable the automation of services housed within Docker images.
Recognizing the need for knowledge transfer, we established a robust knowledge base comprising instructional videos to streamline the process of creating new tests and expanding the framework’s capabilities.
Our testing efforts extended beyond our immediate scope, encompassing the configuration of related services and necessitating seamless communication with distributed teams across different time zones. And last but not least — despite the pressing time constraints, we completed the project within the specified deadline.
Infrastructure: Jira, Confluence, MobaXTerm, Postman, VRA, Maven, Gradle, git, TeamCity, TestRail, Stash, Oracle Sql, Couchbase.
Stack: Java 8, Bash, Groovy(basic), SQL, RabbitMq, Zookeeper, Kafka, Docker, JSON, YAML, Regular expressions, WireMockServer, Lombok.
Test automation libraries: Junit5, TestContainers, SpringBootTest, Rest Assured.