Founder and Team Mediation on Processes and Technical Depts of Industrial Security Product

Project summary

A company that develops a complex integrated software system for the control of engineering equipment at industrial and civil facilities applied to us. The system it develops is used to collect, monitor and dispatch data received from physical devices connected to it (controllers, sensors, circuit breakers) via an industrial protocol.

The system had already been used at several commercial facilities. The company’s management set the task of increasing sales, which meant constant accumulation of business requirements for the system, as well as the availability of stable versions of the application that could be demonstrated to potential customers.

At the same time, the team constantly faced problems in the development process that could critically affect the achievement of the set goals. The team and the management mentioned the following concerns:

Constant failures to meet the delivery date. As a rule, about 30% of all the planned issues were implemented by the announced deadline.
The team had no understanding of the task completion criteria. There was no concept of a full-fledged high-quality release, they did not use the Definition of Done at all.
Constant arguments and unnecessary conversations at daily stand-ups took a lot of time. As a result, the team refused to hold them.
When delivering the system to customer facilities, the implementation department used a lot of manual labor − the system was assembled, deployed and updated manually.
The employees did not always understand what goals the company has, what it plans regarding product development − this information was not communicated to the team.
The team members did not understand their areas of responsibility − who and for what should be responsible, regarding the position, project, direction. Being the project manager, the employee also performed other duties on the project.

As part of consulting, JazzTeam held many events with the company’s team and management. A strategy for full, detailed immersion in the team and the project was chosen. We planned to communicate not only with the top managers, but also with the technical leaders, developers. We also planned live participation in stand-ups and the involvement of the entire team in transformation activities.

Our task was to comprehensively implement best engineering practices at the level of the company’s culture, to instill our values in their work. The team, in turn, must have accepted and applied them in their further work.

Project technologies

Infrastructure: GitLab CI, TestLink, Postman, Jira, Confluence.

Protocols: HTTPs, GRPC.

Project features

In the course of consulting, the following features and difficulties were identified on the project:

A complex technological product. The system has a large codebase and a complex microservice architecture.
It was almost always necessary to set up and configure the system for a specific customer. Large variation of topologies.
The principles of Continuous Integration and Delivery (CI/CD) were not implemented in the development process, the product was assembled manually, which negatively affected the speed and quality of product delivery.
Lack of technological leadership. The company’s technical leaders did not promote the idea of implementing best practices of development, did not defend them before the management.
Team’s insufficient process experience. Weak information connectivity within the team. The backend and frontend departments have different processes.
Lack of an immersion system for new employees. There was no practice of documenting knowledge on the project on a regular basis. As a result, “keepers of knowledge” appeared in the team. Only they knew all the features and specifics of implementing their part of functionality. As a result, there were problems with staff rotation, a long and complex process of introducing new employees to the project.
There was no conscious Test Management. Testing culture was not developing. QA engineers worked inefficiently.
Constant overwork, including work on weekends for almost the entire team, led to burnout and loss of motivation.

Implementation of test automation and CI/CD on the project, as well as transfer of best engineering practices at the level of the company’s culture

Constant failures to meet deadlines and release delays

No practice in using the Definition of Done

Many processes are manual

Each team member’s areas of responsibility are fuzzy

1

Deep immersion in the project and identification of the team’s main problems
2

Development of a strategic transformation plan and its agreement with the company's management
3

Implementation of the strategic transformation plan Conducting daily stand-ups with the participation of the management and the entire team Holding meetings on a regular basis (daily meetings with the management, team technology meetings, one-to-one meetings with all the technical leaders)
4

Setting up the automated testing process and developing first Unit tests for the server part of the application
5

Conducting retrospectives on the project
6

Implementation of Continuous Integration and Continuous Delivery (CI/CD)
7

Transfer of knowledge to the customer's team Creation of all necessary documentation (project console), recommendation of specialized literature for advanced training of employees on the project

The process of Continuous Integration (CI), as well as test automation on the project were implemented
Test management and release management processes became fast and optimized
Regular and high-quality product deliveries to end users are ensured
The team is stable and motivated. Increased engagement and responsibility for the result

Block diagram: Implementation of test automation and CI/CD on the project, as well as transfer of best engineering practices at the level of the company's culture

Consulting format

After deep immersion in the project and identifying the main problems of the team, we developed and applied the following consulting tools:

Daily personal advice to the company’s management.
One-to-one meetings with the technical leaders.
Team meetings dedicated to major technical issues.
Conducting a team retrospective.
Involvement of a manager to implement the strategic transformation plan.
Regular involvement of the consulting team’s head to analyze the effectiveness of the improvements made.

In the process of consulting, a strategic plan of transformations was developed, which was presented to the company’s management:

Elimination of all possible bottlenecks. Regular stand-ups and retrospectives. Participation of each engineer in estimation. Creation of a document containing the company’s vision related to important issues of product development, which is available to each team member.
Recognition of problems by the whole team, readiness to solve them. Introduction of the decomposition and estimation culture. Gaining the support of the technical leaders. Implementation of work with technical debt.
Introduction of automated testing, CI/CD.
Implementation of task performance criteria. Demo format reinterpretation. Using the Feature Teams approach.
Creation of a professional QA engineering department. Test Management.
Learning to accumulate knowledge, fix it, create a training system. Reducing the time new employees need to immerse in the project. New employees must absorb the value approach.
The team reaches the level of self-organization, when it independently searches for solutions to problems in the intended direction. Restoring confidence between the management and the engineers.

Process of implementing the strategic transformation plan:

Daily stand-ups with the participation of the management and the entire team, including the implementation, analytics, design and testing department, were introduced from the first days of consulting. At stand-ups, one should not be afraid to put the team in an awkward spot, to show leadership skills.
Holding meetings on a regular basis:

Daily meetings with the management.
Team technology meetings − dedicated to the project architecture, CI/CD processes, automated testing, discussed the implementation and application of topology virtualization, answered the questions of analysts on the project.
One-to-one meetings with all the technical leaders. The focus of development for each attendee was formed.
We held a meeting with the management. It was dedicated to the employee incentive scheme.

Development of first Unit tests for the server part of the application. Demonstration to developers and managers at stand-ups that it is within the team’s powers. In general, defending the need to create different types of autotests.
Conducting a retrospective. A guide for the retrospective facilitator was provided so that the team could conduct retrospectives independently in the future.
Development of a strategy for the implementation of CI/CD processes.
A project console was introduced to quickly access the most important information. We created documentation containing a description of the project, which will help new employees quickly immerse in the project.
Professional literature on various topics was recommended − project and process management, testing, development, CI/CD, etc.

Results and achievements

The most important result of consulting is meetings with the company’s management and the team held by the head of the consulting team. All records were handed over to the company and added to the final documentation, including transcriptions of the most important meetings. In addition:

The team switched to the iterative development model with each sprint planning − all tasks are evaluated before planning, estimation is applied, and large tasks are decomposed.
Increased transparency of processes within the team through the introduction of stand-ups. A checklist for the stand-up facilitator was developed. Stand-ups are held in a live format with excellent timing (~30 minutes) for a team of more than 20 people.
We changed the format of demonstrations based on the sprint results − now the whole team participates in the demo, the results are demonstrated by the developers. The team’s involvement and responsibility for the result increased.
A project console with all the necessary project documentation was created and is constantly updated, thereby accelerating the process of introducing new employees.
The “Task Readiness Criteria” document was developed and introduced. As a result, we solved the problem of the concept of task completion. A checklist with criteria is added to each task. Failure to comply with the points from any criterion is highlighted at stand-ups and then returned to the developers. QA engineers carry out continuous monitoring of criteria fulfillment.
We introduced writing Unit tests by the team of backend developers on an ongoing basis, contributing to the detection of bugs and system instabilities at the earliest stages.
After each sprint, the team recorded metrics for changing the code coverage, constantly monitored the growth trend, thereby motivating the team. Two key parameters are used as reference metrics: general code coverage and code coverage of the business logic module.
The role of the QA engineer on the project became more important. Participation in sprint planning, demonstrations, control over the task completion criteria. We introduced a preliminary analysis and testing of requirements by QA engineers before transferring to work. We implemented the practice of using test documentation and test management systems.
Automated deployment of the client and server parts of the application to work benches was implemented. Versioning was added to releases.
Thanks to the retrospective, we identified, processed and proposed a solution to the most exciting issues, including the issue of constant overwork.