An AI-based Approach to Cost Reduction in SDLC

Copied!

Introduction

Arguably, resolving bugs and security issues in the early stages of the software development life cycle (SDLC) can save costs in the long run. This is because the cost of fixing defects and security vulnerabilities increases significantly as the software progresses through the SDLC.

But how do we calculate the overhead or how can we calculate the cost savings after a shift-left model applied to quality improvement?

This article introduces a methodology driven by AI that aims to reduce the cost of SDLC (Software Development Life Cycle). The approach is based on a machine learning model that utilizes the results of source code scanning..

When faced with an unfamiliar codebase, our approach involves four key steps. Firstly, we identify and prioritize the “pain” points, or the areas of code that are likely to lead to high development and maintenance costs. Secondly, we evaluate the alignment of the development team. Thirdly, we establish a “quality target” based on our findings. The next step is to initialize the model with an estimate of the incremental cost of fixing bugs during the SDLC phases. Ultimately, we estimate the cost savings resulting from a quality improvement, implemented upfront (sometimes referred to as a “shift-left” approach).

The primary objective of the article is to challenge the widespread notion that source code scanning is merely a “nice to have” method that exclusively concerns developers and drives up development costs. On the contrary, this practice should not be neglected except in the event of major incidents, such as security breaches, catastrophic crashes, extensive downtimes, or when it is required for an M&A process.The integration of the source code in SDLC ensures a cost reduction in the range of 50% or more. Read how this could be achieved!

The first element: The Cost Comparison between Bug/Vulnerability Fixes and Feature Development in SDLC.

Although the claim of cost savings is well known, it is not clear what exactly affects the cost and how the cost is actually reduced and at what level. The main factors that affect the cost of a bug fix and the cost of eliminating a vulnerability in software development can be affected by several factors, such as:

Severity: Bugs can range from minor cosmetic issues (styling) to major functional problems. . Vulnerabilities can range from minor security weaknesses to critical security flaws. The severity can affect the amount of time and effort required to eliminate it.
Complexity: Some bugs are simple to fix, while others may require significant changes to the code. Logical bugs cost more. Some vulnerabilities are straightforward to fix, while others may require significant changes to the code or architecture. More complex vulnerabilities may take longer to eliminate and require more resources.
Stage of the SDLC: Bugs and vulnerabilities that are discovered earlier in the development cycle are generally less expensive to fix than those found later in the cycle or after release.
Availability and cost of resources: The availability of resources, including developers, testers, and tools, can affect the cost.
Code quality: High-quality code with good documentation and clear structure is generally easier and faster to fix than poorly written, messy code.
Communication and collaboration: Effective communication and collaboration between team members can help to identify and fix bugs more quickly and efficiently.
The cost of a vulnerability elimination depends on compliance requirements too: If the software must comply with specific regulations or standards, such as PCI DSS or GDPR, the cost of vulnerability elimination may be higher due to additional compliance-related processes and requirements.
Impact on users and business: The potential impact of the vulnerability on users and the business can affect the urgency and resources allocated to eliminate the vulnerability.
Testing environment: The testing environment can impact the ability to reproduce and diagnose bugs, which can affect the cost of fixing them.

Calculating the cost directly (via algorithms) is nearly impossible due to the wide range of factors that influence it. Therefore, we estimate the cost of bug fixes by considering the productivity of developers and testers throughout the various stages of SDLC. Specifically, we calculate the percentage of resources dedicated to bug fixes compared to the percentage invested in feature development.

We conducted an analysis of 20 commonly used OSS frameworks, in which we calculated the ratio of bug fixes to new feature development. Using Github revisions, we tracked the commitments per user developer over time and classified them as either defect fixes or features. Let us now examine the distribution of bug fixes and features over time for three well-known frameworks.

The ratio of “features” to “fixes” in Odoo is comparatively worse than the other two distributions, whereas Tensorflow exhibits the best ratio of the three. While this ranking is qualitative and may seem obvious to experienced programmers, it is not particularly useful for cost calculations. Therefore, we have utilized a simple AI rule that models an expert’s perspective.

With the aid of this rule, we calculated the M-ratio (maintainability ratio) for a sample of OSS frameworks by analyzing the commitments of the top performers. The M-ratio serves as a useful metric for measuring a codebase’s maintainability and can be easily calculated by analyzing the revision history of the codebase. Our ML-model focuses on reducing costs and improving the M-ratio for software development projects.

A mere maintainability estimation is insufficient to create an effective plan for improving code quality. With limited resources, stakeholders vie for a significant portion. To prioritize components for code refactoring within a product, we can estimate which ones require immediate attention and allocate resources accordingly.

We need to continue working with open-source software (OSS) to ensure transparency in the results. Let’s take the Docker organization’s codebase as a specific example. Initially, it may seem like the team commitments are evenly distributed.

We have scanned 5 repositories, recalculating the same metric per repository. The results are given below.

Based on our analysis, it appears that docker-PY has the lowest level of maintainability. Therefore, it’s important to take into account the overall quality results of the source code, which are provided below.

In a scenario like this, the development team may request additional resources for Python and GO, while the management team may prioritize higher productivity, and the executives may potentially overlook both concerns.

By utilizing a set of straightforward AI rules, we could prioritize a successful refactoring plan that enhances quality and reduces development resources, without the need for additional resources.

Since docker-py is the sole repository where Python developers contribute and its quality is subpar, we should prioritize its refactoring. This would improve the overall quality while also reducing the cost by decreasing the number of occupied Python developers. Similarly, we should prioritize refactoring docker-compose to free up Go-developers who can accelerate development and reduce overall maintenance costs.

The second element: The code quality benchmarks.

Improving code quality can often feel like an endless task, much like the mythological figure of Sisyphus rolling a boulder up a hill only to see it roll back down again.

But when do we know when to stop improving code quality?
Is it necessary to eliminate every detected issue, including code smells and vulnerabilities, or reduce complexity to a certain level?

These questions may seem pointless, but they highlight the need to establish clear goals and priorities for code quality improvement efforts.

Code quality is a temporary result of a set of processes, not an absolute achievement.

We can observe that a codebase appears to be relatively clean and its quality has improved compared to previous versions. We can also compare the code quality of different products, such as Product A having better code quality than Product B. However, how do we draw such conclusions and is it possible to automate this process?

Since code quality is not an absolute metric, its measurement always relies on a standard or benchmark that we consider to be acceptable. The question then arises, what is this standard and what attributes define good software quality? Unfortunately, these questions are difficult to answer and there is no widely accepted standard benchmark that covers all aspects of code quality.

Nevertheless, we can say that a codebase with fewer defects on average is “better” in terms of this aspect. However, it may have other issues such as more vulnerabilities and higher complexity.

Let’s assume that we identify a set of characteristics that could potentially determine the quality of a codebase. For the sake of simplicity, let’s focus on just five characteristics, but it’s worth noting that there could be many more or different ones that an organization might choose to prioritize. Ultimately, the goal is to select the quality characteristics that are most relevant to the organization’s clients, as they will likely generate the most change requests and bug fixes, leading to increased maintenance costs.

The density of Defects: violation of best programming practices
The density of duplications: blocks of duplicated code
Secrets: Hardcoded tokens
Security Vulnerabilities
Number of outdated linked packages

We have selected ten “reference” OSS frameworks that were developed using a similar mix of technology stacks. From these frameworks, we calculated the average quality metrics for the aforementioned characteristics. This average will serve as our quality target, and we should aim to refactor our code until we approach the target quality for each characteristic.

It’s important to note that achieving this level of quality is not a one-time process, and it will require step-by-step improvements.

The third element: The distribution of detected bugs and vulnerabilities over the phases of SDLC

Although we have come across some articles that propose methods to calculate this distribution, we have not yet found an accurate estimation. However, after consulting with ChatGPT4, we obtained a reliable distribution that we will use only for model initialization. It is important to note that several factors can affect this distribution, including the project management approach, the organization’s structure, and the product’s maturity.

We will use the distribution shown below as a starting point, which will be recalculated throughout the software development life cycle (SDLC). The objective of this methodology is to increase the early detection of defects and thereby reduce the overhead of resolving post-release issues.

The fourth element: The cost of resolving a bug or eliminating a vulnerability over the stages of the SDLC

Resolving bugs and security issues in the early stages of the software development life cycle (SDLC) can save costs in the long run. This is because the cost of fixing defects and security vulnerabilities increases significantly as the software progresses through the SDLC.

If defects are identified and resolved early in the development process, it will be less costly and time-consuming to fix them compared to later stages of the SDLC. This is because the later in the development cycle a defect is discovered, the more expensive it is to fix. For example, if a defect is discovered in the testing or deployment phase, it may require significant changes to the code, which can take a lot of time and effort to implement.

Similarly, if security vulnerabilities are identified and addressed in the early stages of the SDLC, it can save organizations from the cost of potential security breaches and the associated legal and reputational damages. Security issues that are left unresolved can result in costly data breaches, which can cause long-term damage to an organization’s reputation and bottom line.

There many surveys present the defect resolution cost over the stages of the SDLC. The cost distribution shown on the intro’s figure reflects a pessimistic estimation much depended on old-fashioned deployment models of products.

Twenty or even ten years ago, distributing new versions and bug fixes was limited to floppy disks or download pages, at best. Today, the majority of products exist in the cloud, significantly reducing the cost of distributing fixed versions. However, distribution models are just one factor that affects overall cost. Other factors such as new version adoption, documentation, training, and API integration continue to negatively impact overall cost.

In this article we will use as a starting point a more optimistic scenario [The Cost of Finding Bugs Later in the SDLC ] . This is an initial hypothesis that should be re-estimated during the SDLC.