As part of the work on the project, we developed and implemented in the already existing system a component for migration of telecom-oriented information on customers’ payments from one system to another with complex transformation of data structure. The complexity of works was in analysis and transformation of the data format and their migration into the system.
This application is a component for transferring and converting billing data about customers from an outdated application into a stably working existing system in the telecommunications industry.
The team had a task to develop a new component and to adapt the existing system to the new file formats containing business data about customers.
The main function of the component is data processing of the following business areas:
Also, the integration of this component with the existing customer’s infrastructure was done.
The application includes several converters for different file formats:
For processing XML files in a multithreaded environment, the “pipelined data processing” approach was chosen and several Pipeline Processors were implemented. Special XPath query language is used for searching the necessary information in the XML file.
Data sampling from XLS files is carried out using Apache POI, a popular library by the Apache Software Foundation.
JSON files are used to store the reference data, and they are converted into a Java model using the Gson library. Each of the converters extracts data from the source file, converts it into a special Java model and transfers it to the following processing.
For the component integration with the customer’s infrastructure the additional events and their processors were created in the existing system.
From the very beginning of the development the entire component code was covered with the Unit and Data Driven tests, that allowed us to make stable the development process of the new functionality and to be sure our changes would not affect the existing system. Load tests were written to exclude possible memory problems, debug the component on a large number of loaded files, since the customer downloads more than 25,000 files simultaneously in a multithreaded environment.
Technologies: Java, Maven, JPA (Hibernate), JDBC Connection, Spring, Quarz, JAXB, Xpath, Apache commons, Apache POI, Gson, JUnit, Data Driven Tests, SVN, XmlSpy, IntelliJ IDEA.
During the work on the application, we have done a lot of work on research, implementation and directly on the integration of the component:
The development process was divided into weekly sprints, at the end of each sprint there was a demo with the customer’s participation.
