Development of a service for the communication with the cloud IoT platform

Provided services

Client

A data transmission operator in the IoT (Internet of Things) field having its own network of base stations.

Product

A service for the communication with a cloud IoT platform for data collection and energy metering devices (MD) management, which allows users to obtain data from current devices, configure their operation, and connect new devices.

Business challenges

To create an innovative popular solution for data acquisition and managing energy metering devices for big energy suppliers (as well as housing management services (HMS), manufacturers, integrators, and service companies), which will be unique in the region market. Within the framework of the project, JazzTeam engineers worked closely with the customer’s R&D department. Our specialists were involved in the project as an independent team, whose initial task was to create the product’s MVP (a service for the communication with the IoT platform) based on the existing solutions and technical assignment provided by the customer. The accomplishment of this task required JazzTeam specialists to perform in-depth research and quickly master a very complicated domain area.

Diagram of the system components and the connection between them involved in the process of collecting data and managing energy metering devices for large electricity suppliers

Project’s technological features

The customer’s company owns and operates a cloud M2M telecommunication platform (ensuring machine-to-machine communication) and a network of base stations. The base stations are located in many regions. Each base station receives data packets from metering devices via radio modules connected to them and transfers data to the M2M platform. The communication between the metering devices and the base stations is carried out via wireless connections that use LoRa spread spectrum modulation.
JazzTeam specialists were responsible for the development of several components, based on which it was necessary to implement a service for the communication with the cloud IoT platform for energy resources accounting. With the help of this service product users should receive data from the available devices, configure their work, and connect new energy resource meters.
One of the conditions that had to be taken into account during the product development was the limitation of the transmitted data packets’ size. This was reasoned by several factors and specifics of communication between the stations and the cloud platform. First, GSM traffic costs a lot. Second, there was a risk of losing a large amount of data: if a station is transmitting data, it cannot receive it at the same time.
The necessity to work with data at a rather low level: to ensure data transfer, it was required to parse the transmitted information into bytes. However, Java language, which was used for the system development, is intended to work with high-level data structures (objects), not bytes. Therefore, to implement the system, it was required to additionally study and use third-party libraries.
Testing was performed by the customer, our team didn’t have direct access to the system’s end users, so we had to resolve all the issues through the customer, which often slowed down the process.

Project’s features

The necessity of rapid immersion in a complex specific domain area.
Frequently changing requirements for the minimum viable product (MVP).

To develop a web application for the online signature of documents

Cross-browser and cross-platform compatibility

Minimal changes on the server side

Providing effective support of the solution

1

Conducting research to solve the main project challenge Testing and controlling the applicability of different approaches to the stated application requirements
2

Developing a web application Applying HTML5 best practices and standards
3

Optimizing the load on the backend Splitting content into files on the client side in the browser, improving encryption and signature functionality
4

Ensuring the cross-browser compatibility of the application Testing the application taking into account all required conditions and options

The cross-browser solution for the electronic signature of documents based on HTML5 technology was successfully implemented

Development of a service for communication with the IoT platform for energy resources accounting and management of energy meters

Approaches and solutions

Creating the product’s MVP based on the customer’s existing solutions. During the MVP development, we checked several hypotheses put forward by the customer. At the first stages we prepared the product version with 2 functions: remote meter shutdown and transfer of 1 byte of data to the device. At the same time, we performed research on the necessary frameworks, libraries and were gradually expanding the functionality. As a result of the research, the team was able to find solutions that made it possible to achieve the goal — to control a meter remotely. We created a prototype that demonstrated the general principles of the system’s implementation. The most important task was the strategically correct choice of the most relevant and successful out-of-the-box solutions (libraries, frameworks).
Research-based creation of the system’s components. Next, we implemented all required parts of the project’s back-end in full. We used Swagger framework, which allows visualizing the system’s components for a better presentation of the completed work to the customer. Our team developed the following components for data conversion, additional acquisition and analysis:

The «SNMP agent» which enables connection to the cloud telecommunication platform (M2M) and control of devices via the SNMP protocol. We implemented the support of obtaining and setting the dynamic parameters of IoT devices obtained via gRPC in accordance with their identifier. All agent commands were described in a MIB file, logging to various sources, including syslog, was added too. For the SNMP agent implementation, we decided to use the existing SNMP4j library. The developed agent was successfully incorporated in the customer’s deployment system as a Docker container.
A «Protocol Converter», which was intended for the standardized exchange of data on energy metering devices’ readings (water and electricity) with external applications. The converter converts messages from the standard used by external systems to the internal standard and vice versa. At this stage it was necessary to implement parsers for binary files and protocols (including due to the peculiarities of the meters’ operating and Java programming language). The M-BUS protocol is used for sending all messages. Parsers for different standards were created with the help of Kaitai declarative language for describing binary formats and data structures. During the implementation of parsers, our team discovered and fixed critical bugs in the Kaitai fork. Unit tests were created for the converter’s entire functionality.
The prototype of the «Module for re-requesting missing meter readings». The module receives readings requests from external systems and checks the presence of these readings in the database. If they are not available in the system, the module forms a request to take readings from the metering devices. When generating requests, the module determines whether to make single readings requests or a request covering some period. This functionality is designed to minimize requests to the base stations (which prevents data losses).

Implementation of the RESTful system based on the previously created components. The service is a system consisting of the set of REST services for the communication with the MDs, which makes it possible to implement any client part in the future. The system has a large number of features: it allows adding and deleting meters, taking readings, etc. Since it consists of microservices, it is possible at any time to remove one service and connect another one, and change a protocol (for example, from DLMS to POS), while the rest of the functionality will work stably. This provides great opportunities for developing the client network and cooperating with new companies (which will not require significant modifications of the system).
Data exchange between the client and server is performed in the JSON format. The system uses the PostgreSQL database. The database contains tables that are used to store data about devices, readings, users, and other necessary information. Communication with the database is performed via Hibernate and JPA.
From our side, the project included back-end developers, who at a certain stage undertook the task of creating the service system’s UI. The existing solutions and tools (BootStrap, Lequery) used by the customer’s team were taken as the basis. Also, a JazzTeam QA specialist joined the project.
We applied the best practices of setting up and managing processes: application of Scrum approaches and activities, professional use of Jira. Our team worked within the culture adopted as a rule at JazzTeam. The development was done in stages, in short iterations, the project progress was constantly monitored at intermediate demos. Our team took up a proactive position, constantly maintaining contact with the customer, initiating open discussions of all problems and nuances. Best practices for setting up and managing processes on the project were shared with the specialists with whom we worked on the project. This approach was appreciated by the customer, who noted the high level of professionalism of JazzTeam management.

Results and achievements

The system was implemented on time, deployed and began to be used by the customer already at the development stage. The product we created made it possible to transfer data over long distances with low energy consumption. It allows prompt receipt and processing of energy resources meter readings, which results in lower operating costs for energy companies and increases the efficiency of their operation. The work of our team resulted in the implementation of the following system’s features: automatic and manual setting of metering devices, management of readings and additional acquisition of missing meter readings, control of relays and MD limits, ability to track MD events by recording all events in the MD log, tariffs management in the MDs, electricity quality indicators management, setting and managing time and managed tasks in the MDs. The customer was able to significantly increase sales of the cloud platform and continues developing its partner network.
The team found and fixed critical bugs in the Kaitai fork, which resulted in costs reduction on the customer side without compromising the project’s quality and timing.
The application works stably. The Unit tests written by our team cover the entire product.
A transparent Agile culture based on sincerity and honesty was built in the process of work, which made it possible to achieve high results in a short time. During the cooperation, the customer was fully involved in the development process and was always aware of the status of certain tasks. Thus, due to constant feedback, our team was able to change the system in the shortest possible time to meet the necessary requirements.