40% Less Bugs through Manual and Auto QA Standartization for IoT Product

The project represents the development of a service intended for the interaction with utility metering platforms. It processes readings received from metering devices and provides remote access to them. With the help of the service, platform users can receive data from metering devices, find out the amount of energy consumed, monitor the dynamics of costs, remotely control equipment, and change the settings of metering devices.

The service is a RESTful system consisting of a set of microservices. Microservices are a layer between end users and metering devices. They allow receiving data, as well as controlling metering devices via HTTPS, DLMS and IEC-104 protocols.

You can get more information about the technical features of the project in the previously published material.

At the beginning of the project, our company provided a service that included MVP development and professional project management. Over time, when the product grew from an MVP into a full-fledged product, we offered the customer additional services to ensure the product quality. JazzTeam specialists were assigned the task to establish the process of product manual testing, and then add the process of automated testing on the project. The processes included the following tasks:

Continuous black box testing and UI testing were implemented to check that the user interface complies with the established standards and regulations defined by the customer. Such checks consist of validating user interface objects that are directly displayed to the user when they interact with the application services. To do this, various test design techniques are used:

Such testing allows maintaining the quality of the product, finding bugs with no need to wait for regression tests and helps check most of the user actions, the interaction of services and components.

After preparing and organizing regular black box testing, we implemented integration testing. It allows detecting bugs that occur when microservices interact with each other, as well as bugs that occur when connecting the user to metering devices through microservices. When testing, a special GuruxDirector tool is used (an open source solution for working with DLMS devices), which allows reading objects (the object is the main element of the metering device information structure, contains all parameters and data, and also has a unique logical name) from electricity, gas or water meters that are compatible with the DLMS/COSEM protocol. GuruxDirector allows checking for bugs related to the interaction of several microservices when reading objects from metering devices.

Another peculiar feature of providing product quality assurance services was the introduction of REST API testing to check the operation of functions that are not available on the UI, as well as conduct additional tests to check requests sent to metering devices. This check allows making sure that the interaction of microservices with each other is stable and data is received through microservices using HTTP methods.

Moreover, during the development of a new authorization microservice, the manual QA engineer carried out work, which consisted not only in testing the new microservice, but also in drawing up a document describing the updated authorization service, as well as creating various types of authorization matrices depending on the user’s role. This information is placed in the customer’s Confluence and added to the technical requirements for the product.

As part of release activities, regression and smoke testing were introduced on the project.

To carry out these types of testing, the following features were agreed upon and implemented:

These types of testing help to make sure that the main functionality of the application after new version release works according to the requirements and does not have side bugs caused by new changes, which allows finding bugs on various production installations before end users do it.

Our company started the process of implementing automated testing from scratch. Jointly with the customer, a list of critical checks to be covered by autotests was developed and approved. We collected all the necessary information, developed a work plan, decomposed the tasks, and estimated the amount of work hours within the initial stage.

The initial task was to develop a framework for the automated testing of critical product functionality. Test cases were selected that affect the operation of the UI, the work on downloading and checking the reports generated by Excel and on checking the operation of metering devices through the GuruxDirector tool. To automate these test cases, Excel file clients and the Gurux client, which simulates the work of the GuruxDirector tool, were created.

The Excel client is developed on the basis of the Java Apache POI library, which allows reading Microsoft Office documents, in particular MS Excel. In the application, when requesting a report, it is downloaded as a .zip archive. The Excel client unzips and extracts the Excel file. Next, a Workbook object and an Excel Sheet object are created, and line-by-line reading is performed from each sheet line into the Map data structure object in the Java programming language (Map<Integer, List<String>>). After that, the data from the data structure object is distributed to the desired report object. To obtain the final result, the data in the created report is compared with the data from the database. The results of the check are considered positive if the data match and there are no duplicates of metering devices in the report.

The GuruxDLMS client is developed on the basis of the Gurux library and goes through the following steps to obtain data:

Within the framework of the Gurux client, a test case was automated using the multi-threading property to check the simultaneous operation of several metering devices.

Another non-trivial, but very interesting task for automated testing was the development of a framework for monitoring the work of the customer’s product using data collected by the Prometheus monitoring system.

The Grafana tool is used to visualize the data from the monitoring system. The customer set the task of creating autotests for the selected product metrics for a certain period of time that meet certain conditions. Before starting the development of the framework, the QA automation engineer studied the environment of the customer, the form in which the metric data is stored, and how it is possible to obtain data for a period of time for a specific metric. The framework uses REST API requests using the Rest Assured library to monitor the performance of the product. A query is sent to the Prometheus timeline database to retrieve data on specific metrics for a specific period of time. The test is considered passed if the values of the metrics meet the specified conditions.

Databases: pgAdmin.

Backend: Java 11, Swagger, Grafana, Apache POI, Log4j2, Lombok, Gson.

Test Automation: JUnit 5, Selenide, Rest Assured, Allure, Awaitility.

CI/CD and DevOps: Maven, TeamCity.

Languages, Protocols, APIs, Network Tools: DevTools, JWT, Putty, Gurux DLMS/COSEM Director.

Software Engineering and Management Tools: TestLink, Jira, Confluence, Git, IntelliJ IDEA.