This article will consider the initial stage of CI/CD implementation in a project having a long history. The purpose of this stage is to carry out a status study and develop an implementation plan.
Let us say a few words about the project: it has been developing quite successfully, and today the application confidently occupies its niche in the market. However, in addition to a large amount of code, an equally large amount of technical debt has accumulated over a long time of development. Due to the lack of proper automation of product delivery, the customer’s company experienced difficulties with application delivery scaling. For example, when adding new clients, it was required to manually configure the plugin system, which took much time and caused a lot of errors.
It should be noted that the customer’s team did not have an understanding of how important it was to work with both the technical debt, in general, and CI/CD, in particular. There was no auto-build in the project. In addition, the customer had a negative experience in the implementation of CI/CD: two years before the involvement of our team in the project the customer’s key developer tried to implement DevOps practices, but nothing concrete was implemented. As a result, the situation only worsened, as the team developed a learned helplessness regarding possible implementation of CI/CD practices.
Since the project did not have CI and automated tests, testing and deployment were carried out manually. As a result, the following problems arose:
- The developers did not create and run Unit tests on a regular basis.
- There were no UI automated tests at the test automation level.
- Testing was carried out manually, while there was no test plan.
The problem was on our side as well. At the start of the project our team engineers were focused on very fast inclusion in the customer’s processes, and therefore we were not able to implement the best DevOps practices right from the start. From the point of view of management, we did not pay due attention to this issue too due to various reasons. First of all, the customer did not have a pronounced desire to deal with this issue.
As a result, we decided to implement CI/CD as a separate internal project. So, our company acted as the driver of this process. Regardless of the limited budget an auxiliary manager responsible only for the implementation of CI/CD was hired for this project. In addition, a novice DevOps engineer was involved in work on the project during 1.5 months. In the course of the internship the DevOps engineer could develop a technical assignment for CI/CD system implementation.
It is important to note that the main thing in CI/CD system implementation is the team’s attitude and focus on the results, but not strong technical competencies. Neither the novice DevOps engineer nor the auxiliary project manager had any experience in CI/CD implementation in projects of this type. Without enduring great losses, taking small steps every day and having a clear mindset for success we managed to show that CI/CD implementation is not just a matter of expertise, it is also a matter of motivation and culture.
Preparation for CI/CD implementation
First, we developed the following project documentation:
- Environmental analysis – in this document we list all available hardware and software that are the result of the team’s work.
- Deployment diagram – in this diagram we show what software is installed, on what hardware it is installed, and how it is interconnected.
- Interaction process diagram – in this diagram we show how the software testing and delivery process is organized at the start of the project.
- CI/CD integration diagram – in this diagram we show how the process of testing and delivery will be organized after CI/CD implementation.
- Atomic operation diagram is an auxiliary diagram in which we split all the operations within CI/CD into small atomic operations – building blocks that were later used to create pipelines.
We will take a closer look at this documentation below.
After documentation development the DevOps engineer proceeded to practical proof-of-concept works. The following was done within this task:
- Fully configured test equipment represented by two virtual machines. The setup involved project environments copying.
- Jenkins was installed and configured.
- Basic elements of the deployment process were configured.
The results of the preliminary preparation were submitted to the project team for introduction in the workflow. At the same time, the auxiliary manager helped with the project until CI/CD was implemented in full. Particular attention should be paid to the following points:
- CI/CD implementation should be started immediately after the initial analysis and technical assignment development. Since, firstly, there is a risk that the implementation of CI/CD will remain a technical debt, and the team will get the state of learned helplessness, and secondly, the implementation of CI/CD is a long process, and the result will be seen later, only when there is a critical amount of updates. Therefore, it is better to immediately allocate time to work with the technical debt and after a while this approach will pay off in a big way.
- It is recommended to implement CI/CD in small increments. Then the team will be able to feel the result, get the necessary experience and quick feedback, which will allow achieving the best result.
- The experience gained within the framework of another project was applied in the course of work. A demo on the use of Groovy scripts in CI/CD was carried out on Jenkins. This is the strategy of our company – the experience gained during one of our projects is summarized and transferred to other projects, which allows using the expertise not only of specific project engineers, but of the entire company as a whole.
Auxiliary project manager feedback:
The approach itself – engaging an auxiliary manager to the project – paid off in full. Since the main project manager shall keep focus on important current activities: on-time deliveries, work with the backlog, work with the team, etc., the technical debt is often put aside. Especially, if the workload on the main project manager is too high. The task of the auxiliary manager is only to help the main manager with the organization of work to solve the technical debt.
Therefore, my key task on this subproject is to be the driver of the process, and if the focus on the process of technical debt liquidation is changed, to return that focus. This ensured the continuous progress of the team on this task.
Also, an important part of my job is to ask questions that require to look at the problem from the right perspective and ensure progress in small increments. The project is distinguished by very short delivery time, so releases take place every week. Naturally, in this case, even knowing the ultimate goal, you can easily lose motivation, because the process of CI/CD implementation is not fast. The implementation strategy through solving small problems and posing specific questions helps here. For example, install Jenkins, add automatic Unit test run after each commit, learn how to stop and start the application on the server, etc. And ensure constant progress towards the goal with such increments.
There is a Japanese saying that moving fast is moving slowly but constantly.
Main project manager feedback:
CI/CD implementation in a long-lived project requires significant management efforts. This is especially true if there are no dedicated DevOps in the project and the team does not have a lot of experience in CI/CD implementation. An additional (independent) manager who only focuses on CI/CD implementation is a very good approach in this situation.
This diagram is necessary so that the team, manager, and CI/CD engineer clearly understand how all the elements of the project are deployed. In the process of diagram creation, the team was forced to ask the customer questions to clarify all unclear issues. This is a very important point, since long-lived projects have their own traditions in processes, and few people understand exactly why each action is necessary. There is an instruction and it works, so you need to understand why this or that process emerged and how it works to change the approach to software delivery.
This document is required for the team to work out in detail the environment in the project and figure out which of the existing services, hardware or repositories can be reused for CI/CD implementation and which ones need to be created from scratch. The difference from the deployment diagram is that in the first case we describe all software components of the application, and in the analysis of the environment we describe all the rest components: service providers, hardware, etc.
The second objective of creating this document is to ensure that there is a complete understanding of the existing situation, restrictions that exist in the project.
In any project we always analyze the environment according to the following structure:
- Service providers: hosting, data storage, other cloud services, etc.
- Hardware available: which servers are available in the project in this case.
Important software elements when deploying a new product version:
- Main application directories.
- Version control repositories.
- Database configuration update script.
Important organization elements when deploying a new product version:
- Jira – how exactly the task tracker is used when deploying a new version of the product.
- Database backup storage system.
The list shall be complete in terms of system update process key components. It is extremely important. Each item shall be accompanied by a description:
- What the element is.
- What purposes it serves.
- What risks this environment element bears.
Interaction process diagram
This diagram is designed to form a clear understanding of the internal organization of software delivery processes in the project before the implementation of CI/CD. It is important to take into account all the points, since our task is to automate and improve these processes. And if you forget some important point when describing the state before starting CI/CD implementation, it is quite possible that your solution will not be optimal or will not work.
Test interaction process diagram
Live interaction process diagram
CI/CD integration diagram
After analyzing the current situation with the delivery of the product, we split the entire system into atomic operations and combined them into a pipeline. Atomic operations are basic operations that Jenkins can perform on our system, such as shutting down the server, copying data, version control operations, etc. The entire CI/CD process is then built based on these basic operations.
CI/CD integration diagram
It should be noted that other supporting documents were also created within the project. They were optional, but they made it easier to understand and organize the implementation process:
- Test equipment description – as part of the proof-of-concept we tested all solutions using test equipment, so this document describes our proof-of-concept experience, in particular, what hardware, what software and how exactly they should be used.
- History of activities related to CI/CD implementation in the project.
This article describes a very important step in the overall CI/CD implementation process – preliminary preparation. When implementing a large project, high-quality, detailed preparation is the key to success in further work and like the implementation process itself, which will be considered in our next article, it requires almost the same amount of time.