How to implement CI/CD in an existing project without outside help. Part 1: preparatory stage

This article will consider the initial stage of CI/CD implementation in a project having a long history. The purpose of this stage is to carry out a status study and develop an implementation plan.

Let us say a few words about the project: it has been developing quite successfully, and today the application confidently occupies its niche in the market. However, in addition to a large amount of code, an equally large amount of technical debt has accumulated over a long time of development. Due to the lack of proper automation of product delivery, the customer’s company experienced difficulties with application delivery scaling. For example, when adding new clients, it was required to manually configure the plugin system, which took much time and caused a lot of errors.

It should be noted that the customer’s team did not have an understanding of how important it was to work with both the technical debt, in general, and CI/CD, in particular. There was no auto-build in the project. In addition, the customer had a negative experience in the implementation of CI/CD: two years before the involvement of our team in the project, the customer’s key developer tried to implement DevOps practices, but nothing concrete was implemented. As a result, the situation only worsened, as the team developed a learned helplessness regarding possible implementation of CI/CD practices.

Since the project did not have CI and automated tests, testing and deployment were carried out manually. As a result, the following problems arose:

The problem was on our side as well. At the start of the project, our team engineers were focused on very fast inclusion in the customer’s processes, and therefore we were not able to implement the best DevOps practices right from the start. From the point of view of management, we did not pay due attention to this issue too due to various reasons. First of all, the customer did not have a pronounced desire to deal with this issue.

As a result, we decided to implement CI/CD as a separate internal project. So, our company acted as the driver of this process. Regardless of the limited budget, an auxiliary manager responsible only for the implementation of CI/CD was hired for this project. In addition, a novice DevOps engineer was involved in work on the project during 1.5 months. In the course of the internship, the DevOps engineer could develop a technical assignment for CI/CD system implementation.

It is important to note that the main thing in CI/CD system implementation is the team’s attitude and focus on the results, but not strong technical competencies. Neither the novice DevOps engineer nor the auxiliary project manager had any experience in CI/CD implementation in projects of this type. Without enduring great losses, taking small steps every day and having a clear mindset for success we managed to show that CI/CD implementation is not just a matter of expertise, it is also a matter of motivation and culture.

First, we developed the following project documentation:

We will take a closer look at this documentation below.

After documentation development, the DevOps engineer proceeded to practical proof-of-concept works. The following was done within this task:

The results of the preliminary preparation were submitted to the project team for introduction in the workflow. At the same time, the auxiliary manager helped with the project until CI/CD was implemented in full. Particular attention should be paid to the following points:

The approach itself – engaging an auxiliary manager to the project – paid off in full. Since the main project manager shall keep focus on important current activities: on-time deliveries, work with the backlog, work with the team, etc., the technical debt is often put aside. Especially, if the workload on the main project manager is too high. The task of the auxiliary manager is only to help the main manager with the organization of work to solve the technical debt.

Therefore, my key task on this subproject is to be the driver of the process, and if the focus on the process of technical debt liquidation is changed, to return to that focus. This ensured the continuous progress of the team on this task.

Also, an important part of my job is to ask questions that require to look at the problem from the right perspective and ensure progress in small increments. The project is distinguished by very short delivery time, so releases take place every week. Naturally, in this case, even knowing the ultimate goal, you can easily lose motivation, because the process of CI/CD implementation is not fast. The implementation strategy through solving small problems and posing specific questions helps here. For example, install Jenkins, add an automatic Unit test run after each commit, learn how to stop and start the application on the server, etc. And ensure constant progress towards the goal with such increments.

There is a Japanese saying that moving fast is moving slowly but constantly.

Main project manager feedback:

CI/CD implementation in a long-lived project requires significant management efforts. This is especially true if there are no dedicated DevOps in the project and the team does not have a lot of experience in CI/CD implementation. An additional (independent) manager who only focuses on CI/CD implementation is a very good approach in this situation.

This diagram is necessary so that the team, manager, and CI/CD engineer clearly understand how all the elements of the project are deployed. In the process of diagram creation, the team was forced to ask the customer questions to clarify all unclear issues. This is a very important point, since long-lived projects have their own traditions in processes, and few people understand exactly why each action is necessary. There is an instruction, and it works, so you need to understand why this or that process emerged and how it works to change the approach to software delivery.

This document is required for the team to work out in detail the environment in the project and figure out which of the existing services, hardware or repositories can be reused for CI/CD implementation and which ones need to be created from scratch. The difference from the deployment diagram is that in the first case we describe all software components of the application, and in the analysis of the environment we describe all the rest components: service providers, hardware, etc.

The second objective of creating this document is to ensure that there is a complete understanding of the existing situation, restrictions that exist in the project.

In any project we always analyze the environment according to the following structure:

The list shall be complete in terms of system update process key components. It is extremely important. Each item shall be accompanied by a description:

This diagram is designed to form a clear understanding of the internal organization of software delivery processes in the project before the implementation of CI/CD. It is important to take into account all the points, since our task is to automate and improve these processes. And if you forget some important point when describing the state before starting CI/CD implementation, it is quite possible that your solution will not be optimal or will not work.

After analyzing the current situation with the delivery of the product, we split the entire system into atomic operations and combined them into a pipeline. Atomic operations are basic operations that Jenkins can perform on our system, such as shutting down the server, copying data, version control operations, etc. The entire CI/CD process is then built based on these basic operations.

CI/CD integration diagram

It should be noted that other supporting documents were also created within the project. They were optional, but they made it easier to understand and organize the implementation process:

This article describes a very important step in the overall CI/CD implementation process – preliminary preparation. When implementing a large project, high-quality, detailed preparation is the key to success in further work and like the implementation process itself, which will be considered in our next article, it requires almost the same amount of time.