Migration project of telecom oriented information on clients’ payments from one system to another with complex transformation of data structure

Duration: 1 year
Industries: Telecom Industry
Services: Backend Development; Custom Software Development; Test Automation Services; Software Reengineering, Refactoring and Modernization. Cloud Migration
Software Categories and Types: Integration Solutions
IT Architecture Paradigms and Approaches: Test Driven Development (TDD); Data Driven Testing (DDT); Software Frameworks Development
Technical Expertise: Big Data/ETL and BI/Data Science
DevOps Expertise: CI/CD Basics
Technologies: JUnit / TestNG; Java Standard Edition (SE); CI/CD Automation Servers; Jenkins; Apache Maven; Java Persistence API (JPA); Hibernate; IntelliJ IDEA; Hudson; FreeBSD; Apache Subversion (SVN); Java; Operating Systems; Linux; Java Enterprise Edition (EE); Chef; IDE; Build and Dependency Platforms and Tools; Backend; Containers and DevOps Tools; Spring; XML based technologies; XPath; Test Automation; Source Code Management (SCM); CI/CD and DevOps; JDBC; Gson; XMLSpy; Languages, Protocols, APIs, Network Tools; Software Engineering and Management Tools; Java Architecture for XML Binding (JAXB); Other Technologies; ORM & Persistence; Quartz Job Scheduler; Java Libraries; Apache Commons; Apache POI

Team size (3):
1 Project Manager
2 Backend

Project summary

As part of the work on the project, we developed and implemented in the already existing system a component for migration of telecom-oriented information on customers’ payments from one system to another with complex transformation of data structure. The complexity of works was in analysis and transformation of the data format and their migration into the system.

This application is a component for transferring and converting billing data about customers from an outdated application into a stably working existing system in the telecommunications industry.

The team had a task to develop a new component and to adapt the existing system to the new file formats containing business data about customers.

The main function of the component is data processing of the following business areas:

Financial information in telecommunications area,
Detailed information about the company’s structure.

Also, the integration of this component with the existing customer’s infrastructure was done.

Technical project description

The application includes several converters for different file formats:

XML files which contain financial information about the company’s telecommunication costs,
XLS files which contain information about the company’s structure.

For processing XML files in a multithreaded environment, the “pipelined data processing” approach was chosen and several Pipeline Processors were implemented. Special XPath query language is used for searching the necessary information in the XML file.

Data sampling from XLS files is carried out using Apache POI, a popular library by the Apache Software Foundation.

JSON files are used to store the reference data, and they are converted into a Java model using the Gson library. Each of the converters extracts data from the source file, converts it into a special Java model and transfers it to the following processing.

For the component integration with the customer’s infrastructure the additional events and their processors were created in the existing system.

From the very beginning of the development the entire component code was covered with the Unit and Data Driven tests, that allowed us to make stable the development process of the new functionality and to be sure our changes would not affect the existing system. Load tests were written to exclude possible memory problems, debug the component on a large number of loaded files, since the customer downloads more than 25,000 files simultaneously in a multithreaded environment.

Technologies

Java, Maven, JPA (Hibernate), JDBC Connection, Spring, Quarz, JAXB, Xpath, Apache commons, Apache POI, Gson, JUnit, Data Driven Tests, SVN, XmlSpy, IntelliJ IDEA.

Screenshots

What we did on the project, our responsibilities

During the work on the application, we have done a lot of work on research, implementation and directly on the integration of the component:

conducted a studying of the new file format, analyzed the provided information,
held the refactoring of the existing system,
implemented a new functionality, covered the new functionality with Unit, Data Driven, loading and integration tests, conducted testing at all the development stages,
created project documentation.

The development process was divided into weekly sprints, at the end of each sprint there was a demo with the customer’s participation.

Project features

Work with distributed teams.
The existing system wasn’t adapted for using the new file formats.
High coverage of the new functionality with tests.
An iterative approach was used for the development: work on the Scrum/Agile methodology, daily internal meetings, weekly demos to the customer, using Jira for the project managing, providing with reports and maintaining communication with the customer.

Project results

Implemented new functionality and bugs of the existing system were fixed.
Documentation for the following application support was created.
All project phases were successfully completed, due to using DDT the component was quickly implemented in the existing system and works successfully.

Company’s achievements during the project

The existing system was refactored.
The vulnerabilities of the existing system were detected and fixed.
The business object model was structured, which allows to add new file formats support in a short period of time.
Throughout the entire development period of the new functionality, we dealt with bug fixing of the existing customer’s systems, and made this process stable.