What is technical debt and why can even successful projects face it?
Technical debt is a huge problem for many IT companies. In our practice, we met both small start-ups and large enterprises, which for this reason were in a deadlock and could no longer develop, scale up the product. Moreover, the solution being developed could be quite worthy, employees – proactive, the company – having authority in the market. However, due to the accumulated technical debt, the team absolutely did not control software operation and could not be sure of the delivery quality.
What caused this situation on the project? Technical debt is primarily associated with best practices of software development, testing and delivery, which for one reason or another have not been implemented. At the start of the project, founders and technical leaders often neglect CI/CD, automated testing, Unit testing, building a scalable architecture, product extensibility, the most important engineering metrics, generation of competent management processes, and pay all attention to the rapid development of the product and its entry into the market. After some time, the company should move to the stage of maturity, hone the processes and follow the built development strategy. However, this does not happen, since speed is important for the team and on a regular basis it faces difficulties and bugs. The problems that arise are solved symptomatically, while the cause remains unresolved (for example, delays in deliveries are solved by hiring additional specialists or overtime work, but not by the step-by-step implementation of CI/CD).
At the beginning of any project the lack of development and testing best practices is not so dramatic and distressing, but over time, as the number of components and interdependencies between them increases, control and stabilization of software operation require more and more effort and time. Accordingly, the development of the company in the commercial sense stagnates.
We prepared this article specifically for founders of IT companies, including those ones who do not have a deep technical background. Here, we will explain why the issue of technical debt is so important, and help determine whether your project has problems related to it. If such difficulties exist, you can take advantage of our plan to pay off technical debt, as well as get valuable advice on changing the culture of the team.
How do you know if there is a technical debt problem on your project? Checklist that will help check the situation independently
To simplify the task of identifying technical debt problems, we systematized the experience of working with various companies and prepared a checklist that will help you realistically assess the current situation on the project. Answer only nine questions to understand what technical debt exists on your project and requires focused attention and study.
Impact of technical debt accumulation on the company
Technical debt has global implications for the company. Especially if the founders ignore existing problems, do not invest in working on them, and do not bring up the current situation in the team for discussion. Most often, the following difficulties emerge on the project over time:
- Due to constant bugs, the product becomes low-quality and unstable, which interferes with its implementation and entails reputational risks.
- Product development becomes too expensive and labor-intensive, respectively, the development of new features stops. This makes the solution uncompetitive.
- The team is not sure of the quality of their software, so the founders cannot plan to increase sales and enter new markets.
- Engineers acquire learned helplessness because they do not believe that they can reverse the negative trends, and therefore that the product has a great future. This leads to burnout and diminished performance.
- A serious conflict and tension between the founder and the team develop: due to constant problems, they do not trust their employees and cannot plan further business growth.
As practice shows, even if at the initial stages the team manages to contain the influence of negative factors, over time, as the critical mass of technical debt accumulates, there is a sharp decrease in motivation and performance.
How to get started with technical debt?
So, you have realized that there is a technical debt problem on your project. So, what now? Definitely, it is necessary to tune in to a long step-by-step work, during which the team will have to overcome many difficulties and changes.
The first step to solving any problem is to be aware of it. If you have doubts or you constantly hear about problems from the team, then as soon as possible become versed in the topic of technical debt and current development methodologies. Understanding the basic concepts and interrelationships, you will be able to discuss emerging difficulties with the engineers, monitor the progress of tasks, and progress in changes. Even if you decide to hire third-party consultants to solve the problem, we recommend that you still understand the essence of the issue in order to clearly show the team that you are interested in solving the problem of technical debt, understand all the nuances, and take the issue under personal control.
The second step is achieving realism and honesty among team members regarding the technical debt. Both you and your team should recognize that everyone made their contribution to the current situation, but now there is no point in looking for the main culprit. You need to jointly analyze the situation on the project, understand the position of the project, and focus on solving the problem step-by-step.
The third step is assessing the problems of the project and developing a work plan. Use our checklist to determine what technical debt exists in the company.
Once the weaknesses of the project are understood and a plan of working with them is developed, you need to think about changing the culture in the team. Accumulated technical debt is a signal that some processes (both technological and managerial or communication) have been built incorrectly. Make sure that your team always has the opportunity and desire to discuss difficulties, offer more effective solutions, and express their opinion. This will help to make headway in technical debt remediation.
It is also necessary to change the organizational approach to the project. Put on the issue of remediating technical debt in the plan of each iteration. This activity should become mandatory. The whole team from engineers to founders should understand that certain amount of time per week is allocated specifically for technical debt remediation. It should not be possible to cancel or pause this task.
Another important component of technical debt remediation is the availability of a leader who has expertise in this area of work and will be able to direct and manage the team. We will tell you how to choose such a specialist and make sure of their professional suitability in the next section. You can also hire a third-party consultant.
Finally, one of the most important components of successful work on technical debt is belief in success. Although the nature of the problems is serious, it does not mean that such problems can be solved by only highly-qualified employees. A large number of routine issues can be regularly solved by mid-level specialists (Middle and even Junior+). Moreover, the latter has no fear of failure, and often you can rely on them if there is a powerful strategy and a good leader. The main thing is that specialists have a clear plan of action, as well as a great desire and motivation to master new approaches.
Why does the availability of an experienced technical leader not guarantee that there will be no problems with the project?
The help of an experienced leader is needed to systematically work with technical debt. However, their presence on the project does not always result in resolving the issue. This situation can be encountered under the following circumstances:
- The leader has no experience with debts and changing the culture adopted in the team.
- The leader does not believe (or does not want to believe) that solving the problem with that debt is one of the most important tasks for the team.
- The leader cannot influence the team: they are unable to explain the importance of the issue, to motivate employees, or to put pressure where it is really necessary.
- The technical leader lacks the experience or ability to convince the founders. Such situations often occur when a specialist comes to the company with little experience and bolsters their career in it up to a senior position. Years later, from the force of habit, the founder does not trust the authority of the employee, as they remember them inexperienced and do not see the need to listen to their opinion. In turn, the technical leader grows, masters the latest approaches, understands many nuances, but their proposals are not taken seriously. Over time, this leads to the fact that the leader gives up trying to change something and acquires learned helplessness.
Thus, having an experienced technical leader does not guarantee that there are no problems with your project. Use our checklist to check this nuance.
Checklist: How do you know if your technical leader can cope with the problem of technical debt?
It is difficult to overestimate the role of the technical leader in solving the problem of technical debt. The final costs and success of the work carried out depend on how they set the course and whether they can convince the team.
We offer to fill out a short checklist that will help determine whether your technical leader is able to overcome the problem of technical debt. Each item has its value calculated in points. Calculate the final score and see the result at the bottom of the page:
- If the project has clear problems, while the technical leader has been working in the company for a long time, this already means that they do not cope. There are often two reasons for this:
- The technical leader acquired learned helplessness. Psychologically, they cannot overcome the difficulties that have arisen. Most often this happens when the founder imposes their style of work on the technical leader and does not allocate time to work on technical debt. The technical leader just gives up and tries to ingratiate himself/herself with the management.
(Yes: 1 point, No: 0 points) - The second option – the technical leader is not a leader by nature. They can voice ideas, but never goes the whole length of them.
(Yes: 1 point, No: 0 points)
- Does your technical leader regularly get time to work on technical debt in various iterations? Do they argue for the need for this type of work?
(Yes: 0 points, No: 1 point) - Can you say that your technical leader is a valued advocate for the automation and implementation of CI/CD?
(Yes: 0 points, No: 1 point) - Do you have a feeling that the technical leader constantly talks about any problem on the project, but does not offer an idea or a plan for its solution? Instead, you regularly hear from them that there is a complex global task on the project, which can be solved only if focus heavily on it for at least six to twelve months.
(Yes: 1 point, No: 0 points) - Does your technical leader have a strategic vision of the project despite the difficulties? Do they convey this vision to the team? Do they actively participate in retrospectives, and share best practices with employees? Does the project have documented processes, guides for new employees, and architectural diagrams?
(to any of the questions listed in this item: Yes: 0 points, No: 1 point) - To be honest, do you defer to the opinion of the technical leader? Can they influence your production plans?
(Yes: 0 points, No: 1 point) - Do you have a feeling that the current problems on the project have not been solved for years?
(Yes: 1 point, No: 0 points)
Results:
- 0 points: your leader is fully ready to work with technical debt.
- 1 to 3 points: there are significant obstacles in the path of your technical leader. You need to think about how you can contribute to their elimination.
- 4 to 8 points: unfortunately, your technical leader will not be able to solve the problem of technical debt.
How to deal with the resistance of the team?
Launching processes aimed at technical debt remediation is often distressing for the whole company. This is the eternal fitness of things because the introduction of important practices requires serious transformations in the team: from changing the culture to transforming daily approaches to work. First of all, this is expressed in the clash of interests of stakeholders. Technical debt remediation requires significant labor costs for engineers, which leads to a temporary decrease in team performance and can slow down the process of creating important functionality for commercial deliveries. This can have a negative impact on sales and disturb stakeholders.
However, it is very important to promote the following truth in the company: the longer you delay the situation and accumulate debt, the greater the problems and financial losses for the whole company you will face in the future.
When the change strategy is approved by the managers, the following difficulty arises – perhaps, you will have to deal with the team’s resistance. Being used to working without stress and overcoming difficulties, having lost faith in changes, specialists can boycott new responsibilities for technical debt remediation. Unfortunately, sometimes there is hard logic and an unconscious subversive activity behind this. Usually, this manifests itself as sabotage or a sit-down strike, when engineers perform tasks for a long time, referring to the lack of time to complete your instructions regarding technical debt remediation. Especially if employees know that the founder does not understand the technical nuances.
Therefore, when faced with resistance, it is very important to develop tactics for interacting with the team, create a set of rules, and fix metrics related to the level of technical debt remediation. Constant demonstration of the results achieved will help to convince and motivate employees.
Of course, you can see that some employees are particularly resistant to important innovations or set other colleagues against them in the process of changing values. In this case, it is necessary to make a choice in favor of the future of the company and decide on dismissing the specialist.
Non-obvious advantages of technical debt remediation
In addition to improving technical nuances, accelerating the development process and reaching a new stage of development, technical debt remediation has three important consequences for the company:
- Objective assessment of project processes effectiveness. Technical debt is always closely linked to systemic gaps in processes. They are especially evident in the process of assessing the situation on the project. For example, if your company does not have a test plan for manual testing, then the introduction of automated testing becomes impossible. The CI/CD process (even if it is implemented) will be inadequate without automated testing. Therefore, technical debt will help you see a large number of other problems of an organizational nature.
- Increased performance and higher degree of discipline among engineers. Technical debt management helps to systematize and optimize the daily work of technical specialists. Regularly working with complex tasks, engineers actively develop their skills, as well as learn to provide a rational estimation. This will allow to create a team of realistic developers over time who are not afraid of challenges and difficulties.
- The possibility to transform even the most complex or problematic product. After analyzing all the problems and developing a plan for working with technical debt, you will get a clear understanding of what is the source of your problems. Gradually moving towards the goals, the team will gain faith in the changes and the product will be able to develop further.
Five tips for founders who decided to start paying off technical debt
Despite the fact that it is impossible to solve the problem of technical debt without the active actions of engineers, you can influence the situation and timely launch the work on which the future of the project depends. Whatever stage you are at now, we recommend following these tips:
Constantly develop your knowledge, study modern development methodologies and try to understand their essence. This will help you talk the same language with the team and objectively assess the progress. Based on our experience, we can say that all the founders we advised had to delve deeply into the value aspects of technical debt. And we are sure that it is possible regardless of your proficiency or technical education.
Get the opinions of technical experts more often. Establish a culture in the company which presupposes that everyone’s opinion will be accepted and handled. An atmosphere of trust will help to get complete information about existing problems and build a strategy for their solution.
Be realistic, no matter how hard it is. Try to honestly answer the question whether your team has learned helplessness, whether engineers have a too casual attitude to their work. Constantly monitor the situation and make the necessary decisions to improve it.
Deal with technical debt even in moments of trial and crisis. Unfortunately, over time this problem will not stop existing, but will only have a greater impact on your processes and team’s performance. 15-20% of developers’ time should be allocated to work with technical debt. This activity should be a mandatory part of sprints.
Remember that the following TOP 4 best practices should always be used on the project:
- Unit testing.
- UI automated testing.
- A complete CI/CD process is used by the entire team.
- Regular retrospectives.
Need help with technical debt remediation? Contact us
Want to get expert support in solving technical debt problems? We specialize in consulting in this area. We will be happy to help with a comprehensive analysis of the project, the introduction of best practices, as well as changing the values and culture of the team. Our managers follow Agile methodology, in which the independence and active participation of each team member is especially appreciated. Therefore, we guarantee that after the end of cooperation with our company, your specialists will be able to effectively independently advance in solving the tasks.
Interesting cases related to technical debt remediation
Case 1 – Product Stabilization through Data Driven Testing
In this case, the entire business logic of the application was placed in complex stored procedures and it resulted in the emergence of technical debt. This greatly complicated debugging and increased the cost of product support. Accordingly, it led to product instability in production. Bug fixing inevitably provoked the emergence of new problems. At the same time, the company did not try to eliminate the cause of the situation but continued an endless struggle with bugs, which did not lead to the expected result. To get out of such a difficult situation, the company was forced to invest in a system solution using the Data Driven Testing approach. And this solution made it possible to get rid of the problem forever.
Case 2 – Consulting on overcoming the technological crisis
In the process of developing a complex knowledge-intensive product, the team faced serious difficulties that impeded further growth. At a certain stage, when the number of the company’s customers began to actively increase, the team found that the product was quite unstable. However, the engineers did not have the opportunity to correct the situation, since the project lacked automation, CI/CD, and Unit testing. It took years to completely eliminate the accumulated technical debt. And only systematic, regular, step-by-step work on the implementation of best practices is allowed to turn the tide.
3 – Mediation of relationships between the company founders
The project for the development of a large-scale enterprise product reached a critical point in the accumulation of technical debt. The lack of automated testing, as well as continuous integration and delivery, hampered software scaling-up. Any attempts to make changes led to bugs in production. The entire team constantly experienced stress when it came to development and delivery. This situation did not allow the company to attract larger customers and increase sales. In addition, serious conflicts were generated in the team, as well as the engineers acquired learned helplessness. Solving the problem required a radical revision of processes, changes in the culture, and harder work on the accumulated technical debt.
Checklist: How to understand that your project has accumulated technical debt?
According to our deep conviction, serious enterprise projects 3+ months old have technical debt. If such characteristics correspond to your situation, then you definitely need to allocate a sufficiently significant share of backlog for tasks related to this problem. If this is not done, the team will come to a state of learned helplessness. Also, over time you will not be able to develop the product and work with a large number of customers.
To make sure that there are no problems with technical debt, fill out our checklist. Each “Yes” answer in this list clearly indicates specific shortcomings that need to be worked on.
There is undoubtedly technical debt on the project if the following conditions are met:
- To update the version of the product on the server, your team performs manual actions.
- Over the past six months, there have been no refactoring tasks in your backlog.
- There are no Unit tests on the project or the level of test coverage is less than 70%.
- In your product, there are parts of the functionality that have been unstable in production for a long time.
- There is no code review on the project.
- You regularly hear from the team at stand-ups that some parts of the functionality cannot be changed or changes will take too much time. You also feel that the team as a whole is not ready for changes.
- The engineers’ estimations systematically do not comply with reality. Tasks accomplishment constantly take more time than it was originally anticipated. Every time you ask why the engineer couldn’t complete the task on time, they tell you that it’s very complicated and non-trivial.
- Your specialists are sure that a Junior or Middle specialist will not be OK for expanding the team. They are sure that they need only a Senior, as the product is very complex.
- You regularly hold team retrospectives where you discuss successes and failures, plan improvements, and really do work related to them.
If you want to get a closer look at the nuances and problems of the situation, fill out the checklist How to Know Your Project is in Danger.