B2B e-commerce platform for inventory and order management, including ERP, CRM and inventory management system. The project is developing quite successfully, the application confidently occupies its niche in the market.
Technical description of the project at the time of the project start:
The system has a large codebase with a non-trivial, confusing architecture and mixed layers of code (business layer, working with the database, controller, UI), which makes it difficult to support. The components of the system are not isolated, due to the high level of connectivity, bugs arise with the slightest code changes. The product is very difficult to scale or implement new functionality.
The principles of continuous integration and delivery (CI / CD) are not implemented, the product is assembled and the servers are updated manually, which negatively affects the speed of product delivery. Development and delivery are intense and stressful for all involved in the process.
Since the project did not have CI and autotests, testing and deployment were carried out manually. As a result, there were no Unit tests, and there were no UI automated tests at the test automation level. Testing was carried out manually, and there was not even a test plan.
- Over a long period of development of the project, in addition to a large amount of code, an equally large amount of technical debt has accumulated. Due to the lack of proper automation of product delivery, the customer’s company experienced difficulties with application delivery scaling. For example, when adding new clients, it was required to manually configure the plugin system, which took much time and caused a lot of errors.
- The customer’s team did not have an understanding of how important it was to work with both the technical debt, in general, and CI/CD, in particular.
- There was no auto-build in the project. In addition, the customer had a negative experience in the implementation of CI/CD: two years before the involvement of our team in the project the customer’s key developer tried to implement DevOps practices, but nothing concrete was implemented. As a result, the situation only worsened, as the team developed a learned helplessness regarding possible implementation of CI/CD practices.
Our team decided to implement CI/CD as a separate internal project. We acted as the driver of this process. Regardless of the limited budget an auxiliary manager responsible only for the implementation of CI/CD was hired for this project. In addition, a novice DevOps engineer was involved in work on the project during 1.5 months. In the course of the internship the DevOps engineer could develop a technical assignment for CI/CD system implementation. Further implementation was planned to be carried out by a team of engineers.
Approaches and solutions
- Creation of the necessary documentation for the project to prepare for the implementation of CI/CD. Our team has prepared and implemented the following documentation:
- Environmental analysis – in this document we list all available hardware and software that are the result of the team’s work.
- Deployment diagram – in this diagram we show what software is installed, on what hardware it is installed, and how it is interconnected.
- Interaction process diagram – in this diagram we show how the software testing and delivery process is organized at the start of the project.
- CI/CD integration diagram – in this diagram we show how the process of testing and delivery will be organized after CI/CD implementation.
- Atomic operation diagram is an auxiliary diagram in which we split all the operations within CI/CD into small atomic operations – building blocks that were later used to create pipelines.
- DevOps engineer proceeded to practical proof-of-concept works. Within this task was fully configured test equipment represented by two virtual machines. The setup involved project environments copying. Jenkins was also installed and configured, basic elements of the deployment process were configured too.
You can learn more about research in the field of project development in our technical article.
The results of the preliminary preparation were submitted to the project team for introduction in the workflow. At the same time, the auxiliary manager helped with the project until CI/CD was implemented in full.
- Implementation of CI/CD process. After the documentation had been prepared, it was necessary to implement the planned changes. This process required a number of researches to achieve full build automation, including providing automatic builds for product executables and implementing LiquiBase for database-level version control. The CI/CD implementation process has been divided into the following steps:
- Implementing an automatic build system of the application. There was a special utility on the project that prepared the data for the build. The developer used this utility to create special template files. After that, he used IDE to create a working build of the application with these files. As part of automation, we implemented the Maven application automated build system, which wrapped the use of this utility and allowed us to make application building fully automated. As part of this stage, Unit testing of the finished application was also added.
- Installing and configuring Jenkins.
- Automated update of the Frontend server within CI/CD. As part of this step, we automated the deployment of the frontend application on IIS server.
- Automated update of the Backend server within CI/CD. We automated the deployment of the backend application to the Tomcat server. The following functionality was added to Jenkins: building a new web application, Unit testing, installing Tomcat, copying the web application file to Tomcat, Tomcat starting.
- Updating customization for specific customers. As part of this activity, we initially made a proof of concept, then finalized the application architecture, and implemented deployment automation. A separate engineer was responsible for this area of work, who performed a full cycle of research work for this system – from writing requirements and the sequence diagram to implementing a ready-made solution.
- Database update. As part of this stage, it was required to update the user databases. The peculiarity of this application is that it has not one database, but a large number of databases with almost identical schemas, since each customer had to create his own database instance. At the same time, some customers also had additional customized functionality, which, among other things, required individual, rather than group, database updates.
- Implementing GUI testing. With each build of the application, all Unit tests are run. At the same time, GUI autotests from the master branch are run every night on the test bench, and they completely check the current instance for possible regression. When all tests are completed, a report is generated, which is analyzed by a QA Automation engineer in the morning. You can read more about the implementation of autotesting in a project in our article.
- Automated deployment to Live server. When all the work on automating deployment and testing was completed, we started trial operation on a test server. The test period lasted two months, during which we tested the operation of all elements of the system. At the end of the test period, we began to use this automation on the Live server. This allowed us to automate the work of developers, thus reducing the likelihood of human error when deploying to a Live server.You can read more about each stage of CI/CD implementation in this project in separate technical article.
Stack: Liquibase, Allure, Groovy.
Infrastructure: AWS, Bitbucket, Jenkins, CloudWatch.
- Thanks to the introduction of CI/CD, the process of the product build and delivery was automated. It became fast, transparent, and secure. That allowed ensuring the regularity of releases.
- Keeping a high priority on commercial work, the engineers were advancing every day in the implementation of CI/CD. That approach allowed us not to lose focus on the commercial task delivery, and at the same time, we consistently moved forward to resolve the accumulated technical debts. Below are some illustrative examples of how critical tech debt was dealt with.
- The operation of the product has become much more transparent and at the same time significantly less time-consuming.
Company’s achievements during the project
- The experience gained within the framework of another project was applied in the course of work. Demo on the use of Groovy scripts in CI/CD was carried out on Jenkins.
- The functionality of a custom update of a group of databases was implemented, which made it possible to work automatically with both test and production databases.
- The frequency of the deploy on the test stand increased 5-10 times, and on Live – 3-5 times.