Provided services
Backend Development, Frontend Development, Test Automation Services, Manual Testing Services, IT Consulting, Software Reengineering, Refactoring and Modernization. Cloud Migration, Software Maintenance and Support Services
Client
An international telecommunications company specializing in developing solutions for monitoring and managing communication costs. With over 10 years in the market, the company partners with corporate clients worldwide, ensuring expense transparency and optimizing telecommunications services.
Product
A comprehensive cost control and accounting system for telecommunications expenses, consisting of multiple interconnected components. It helps companies manage communication costs by providing detailed reports and analytics.
The product includes:
- Client web application – a single access point for viewing invoices, managing the organizational structure, and monitoring expenses.
- Administrator web application – a tool for user management, system monitoring, and report handling.
- Event processing server component – responsible for data loading, processing, updating, and automating various workflows.
This system not only ensures expense transparency but also allows employees to distinguish between personal and corporate communications, enhancing the efficiency of telecommunications services.
Challenge
The project involved long-term support for a complex telecommunications system used by various company divisions. Unlike many similar projects where the primary goal is merely bug fixing, this one had a more ambitious objective: enhancing the product’s maturity by improving development and testing processes.
The client required system maintenance and increased stability and predictability, ensuring controlled support with clear Service Level Agreements (SLA). Instead of the standard approach to technical support, where developers fix bugs upon request, the client expected an active influence on system quality through process automation, CI/CD, and a thorough analysis of weak points.
The system faced instability due to complex interdependencies within stored procedures, making quick bug fixes prone to introducing new issues. Data loading instability caused errors to appear sporadically, complicating root cause analysis, while long bug-fixing cycles-taking up to a quarter per release-hindered product improvements. The lack of automated testing increased the risk of defects, and reliance on outdated PL/SQL stored procedures made maintenance and technology adoption challenging. Additionally, the system had multiple security vulnerabilities requiring in-depth analysis and remediation within strict deadlines for certification. Interaction with auditors was complex, as some findings were advisory and required careful interpretation. Finally, code changes had to enhance security without compromising system stability and performance.
It was crucial not just to address existing issues but to establish a foundation for long-term product improvements without significant changes. Achieving this required the implementation of automated testing, release control, system stability improvements, and a gradual refactoring of critical components.
Project Context
The project involved long-term support and evolution of a complex telecommunications system used by various company divisions. In addition to ensuring product stability, an additional challenge was passing a security audit for ISO certification.
As part of the project, the cost control system had to comply with the OWASP Top 10 – a list of the most critical web application vulnerabilities. However, the audit revealed a significant number of security issues, putting successful certification at risk.
JazzTeam Challenge
For our team, this project presented a unique challenge, as we had to enhance system stability without making radical changes to legacy code. This meant that all improvements needed to be introduced carefully and gradually, without disrupting the client’s business processes.
Key challenges:
- Lack of a structured testing system – we had to develop and implement a robust automated testing strategy, including Data-Driven Testing (DDT)
- Minimizing production failures – this required strict deployment control and thorough validation of updates to prevent disruptions.
- Working with an outdated technology stack, the existing infrastructure limited opportunities for optimization and automation.
Solution
To address the challenges, our team implemented a comprehensive approach to improving system stability, automating processes, and enhancing code quality. We focused on CI/CD implementation, Data-Driven Testing (DDT), stored procedure refactoring, testing optimization, and cybersecurity, which significantly improved system predictability and accelerated update releases.
Key Solutions:
Implementing CI/CD for Deployment Automation
At the start of the project, most deployment processes were performed manually, increasing release times and making them unpredictable. To eliminate human error, we configured an automated CI/CD pipeline using Jenkins.
Our team configured automatic project builds after each code change. Implemented automated deployment with rollback functionality in case of failed releases. We set up real-time notifications for developers in case of build or test failures. Optimized quality control: every change undergoes automated checks before deployment to production.
Using Data-Driven Testing (DDT) for System Stabilization
Previously, testing was unstructured, and many errors only surfaced in production, making the system unpredictable. We introduced Data-Driven Testing (DDT) to monitor the stability of critical system components.
Implementation details:
- Developed a structured system of automated tests based on real system data.
- Created a repository of reference datasets in SQL and XML formats to verify data processing accuracy.
- Created a repository of reference datasets in SQL and XML formats to verify data processing accuracy.
- Introduced automated test cases that replicate production errors, ensuring they are immediately added to the regression suite.
- Automated the testing of stored procedures, which had previously been done manually.
Optimizing Legacy Code and Stored Procedures
The system heavily relied on business logic implemented in PL/SQL stored procedures. Their complexity and lack of structured testing contributed to system instability.
JazzTeam specialists developed a phased plan for refactoring critical stored procedures. Split large, monolithic stored procedures into smaller, modular components for better readability and maintenance. They implemented automated stored procedure testing using reference datasets. And optimized SQL query execution, reducing data processing times.
Enhancing Testing Processes and Quality Control
Before our solutions were implemented, the project lacked unit tests, and most testing was performed manually. This slowed bug fixes and increased the likelihood of undetected issues.
We started with the introduction of unit testing with JUnit, covering critical system modules. Developed automated UI tests, reducing reliance on manual testing. Built a reporting system for application stability, enabling better tracking of weak points and preventing regression. Integrated automated API testing, eliminating errors in system integrations.
Process Improvements
To streamline delivery and ensure system stability, we introduced a series of process-level enhancements across QA, development, and release management.
- Incident and Defect Management. We standardized bug reporting, automated prioritization based on severity, and implemented a fast-track process for handling critical issues, reducing resolution time and improving traceability.
- Cross-Team Collaboration A unified collaboration model was introduced, including regular sync meetings and shared documentation, which improved communication between distributed teams and accelerated onboarding.
- Code Quality Control We enforced mandatory code reviews, unified coding standards, and integrated static code analysis to reduce technical debt and prevent critical issues early.
- Release Management Deployment processes were stabilized through strict versioning, dedicated testing environments, and automated rollback mechanisms, minimizing risks during release cycles.
Development Approaches
In this project, special attention was given to enhancing system stability through automated testing, as unstable modules and complex dependencies within stored procedures made system maintenance and development unpredictable.
One of the key methodologies we implemented was Data-Driven Testing (DDT) – an approach that not only stabilized critical system components but also established an effective regression strategy to prevent recurring errors.
Technological Overview
To address frequent failures in data loading and the complexity of testing stored procedures, we implemented a scalable Data-Driven Testing (DDT) approach.
- Stabilizing Data Load Processes. We automated data load validation using predefined XML inputs and reference output files. Each bug led to the creation of a new test case, which was automatically added to the regression suite. We also introduced Java-based database state generation, reducing manual overhead and making maintenance easier.
- Testing Stored Procedures at Scale. Legacy stored procedures were tested using DDT scenarios based on real input and expected output datasets. This approach enabled consistent comparison of execution results, better coverage, and reduced manual effort.
- Continuous Test Growth and CI/CD Integration We built a growing base of reusable tests to catch regressions and embedded DDT into the CI/CD pipeline, ensuring only stable builds were released. Complex business cases were also covered through structured test flows.
The outcome:
- Greater system stability and confidence in releases.
- Faster detection and reproduction of issues.
- Significant reduction in manual QA workload.
For more information about our Data Driven Testing (examples from commercial projects), please refer to our technical article – https://jazzteam.org/technical-articles/data-driven-testing/
Cybersecurity Leadership
While we were working together on this project, the customer was in the process of obtaining ISO certification. To confirm its compliance with the necessary requirements and standards, the system of accounting and control of the costs of telecommunications services had to be audited by an authorized company.
The results of the audit, which was conducted in accordance with the OWASP Top 10 list of the most common security problems, came as an unpleasant surprise to the customer. It turned out that the project had a significant number of information security problems and risks. Since we were responsible for supporting their product, the customer assigned our team the task of fixing the identified defects.
In practice, the task turned out to be non-trivial and labor-intensive. In order to make the necessary changes to the relatively large code base, we had to dive into the product’s specifics and the nuances of each of the auditors’ comments. JazzTeam specialists not only made these changes, but also manually tested each fix. The systematic organization of the manual testing process played a major role in the success of this task.
As we worked to improve the product, the interaction with the audit firm was also fully transferred to JazzTeam. Our role was to negotiate and agree on the nuances of each issue. Some issues were very complex and not directly related to information security, or were advisory in nature. However, our team was able to fully process the audit firm’s comments and make the necessary improvements. As a result, the customer’s product successfully passed the certification and received confirmation of its compliance with the ISO standard.
JazzTeam dived deep into the specifics of the standards required to pass IT product certification. Not only do we effectively address the issues, we also manage the technological communication with the auditing companies and conduct comprehensive testing of the implemented changes.
Result
Key Project Outcomes:
- New functionality was implemented, and bugs were fixed, allowing the client to expand system capabilities without the risk of destabilization.
- Continuous Integration (CI) was introduced, accelerating update deployments and minimizing production errors.
- Application deployment was fully automated, eliminating manual errors and speeding up version releases.
- Automated UI testing was implemented using xml2selenium, along with unit test coverage for the server-side and Android application.
- The system was analyzed for OWASP Top 10 compliance, vulnerabilities were identified and fixed, enabling successful ISO certification.
- Production system stability was significantly improved, ensuring predictable operation and increasing user confidence.
Additional Achievements:
- Bug resolution time was reduced by 15x thanks to Data-Driven Testing (DDT) and an automated quality control system.
- Release cycles were shortened from one quarter to just one week due to automated testing and deployment.
- Regression testing now takes one day instead of several weeks, allowing the team to implement changes faster with minimal risk.
- Recurring errors decreased by 40%, attributed to the accumulation of test cases in DDT and a strict code review process.
- System performance improved by 30% due to stored procedure optimization and SQL query enhancements.
JazzTeam Achievements
- Comprehensive Project Documentation. We developed and maintained over 30 technical documents, ensuring system transparency and facilitating fast onboarding. These included architecture diagrams, setup guides, and multimedia materials for training new team members.
- Database Optimization. We normalized the database structure, which reduced data redundancy, improved query performance, and minimized the risk of logical errors.
- Automation of Integration and Testing. We introduced Continuous Integration and Delivery (CI/CD), along with a structured framework for unit and UI testing, which significantly improved release reliability and quality control.
- Improved Data Management Workflows. We enhanced processes for automated database population, simplifying migrations, maintaining version consistency, and supporting efficient regression testing.
Screenshots
Technologies
Databases: Oracle, PL/SQL
Frontend: HTML, CSS, JavaScript, JQuery, AJAX, Struts2 tags, JSTL tags
Backend: Java (Servlet API), Spring Framework (Spring MVC, Spring Security), Hibernate, JPA, JDBC Connection, Apache POI, SuperCSV, JAX-WS, Tomcat
Test Automation: JUnit, Mockito, xml2selenium, Data-Driven Testing (DDT)
CI/CD and DevOps: Jenkins, Nexus Repository Manager
Languages, Protocols, APIs, Network Tools: Java, REST APIs, SOAP Web Services
Software Engineering and Management Tools: SVN, Jira
Other Technologies: Ehcache, MutationObserver API, CSP 2.0 (Content Security Policy)
