Measuring and Improving Software Delivery Performance
Key metrics and strategies to measure software development team performance, identify bottlenecks and boost productivity
In today’s fast-paced software development environments, teams constantly strive to deliver high-quality features quickly while minimizing operational overhead. Agile methodologies have become a standard framework for achieving these goals, with regular sprints and frequent feedback loops. Where the product manager creates and prioritizes user stories, while the development team implements, tests, and deploys the features to customers regularly — usually by the end of each sprint for agile teams. Teams often measure performance by tracking completed story points, raised defects, and the achievement of sprint goals.
At first glance, this process looks effective, but it’s important to remember that customers care less about internal processes and metrics like story points; they are more concerned with the features they requested and when they can use them. Before delving into a team’s performance, it’s crucial to address potential bottlenecks that could slow down delivery and waste team efforts.
Delivery Bottlenecks
A development team workflow usually starts by planning and prioritizing the items to work on, implementing those items, code review, feature testing, deployment to a staging environment, release testing, deploying to production, and finally smoke testing.
Unnecessary Workflow Steps Segregation
Segregating a workflow step, such as implementation and code review, might discourage ownership and complicate status reporting. When a developer implements a new feature and sends it for code review, which is assigned to another developer, they might think their job is done — they have submitted their code for review, and might not follow up on it. An enhanced workflow combines both implementation and Code Review steps into a single step: implementation, which is the responsibility of the author to implement and follow up on the code review, before moving an item to the testing step.
Code Review Delays
Although cutting down code review as a standalone step should encourage ownership, code review still there, and it is something that couldn’t be eliminated; as without it, code quality may decrease, bugs are likely to go undetected, and knowledge sharing is limited. But, the more time it takes to perform code review, fewer features are delivered at a given timespan. Therefore, monitoring the average code review time is essential to reduce bottlenecks.
Manual Feature and Release Testing
Feature and release testing, as manual steps, create additional bottlenecks; assume two developers completed a user story each at the same time, where there is one QA — one user story will remain in the waiting state until the QA is done testing the other user story. Although, fully replacing manual testing with automation should remove such bottlenecks, it requires investment, such as infrastructure setup and test data management, that teams might not be ready to. However, replacing one of the manual testing steps, namely feature testing, with unit and integration tests, and automatically running those tests through CI (Continuous Integration) jobs, will reduce delivery bottlenecks and provide feedback earlier to the developers whether their changes broke the existing system or not.
Infrequent Releases
Automating a manual step, feature testing, should expedite the delivery only if releases are tested and deployed frequently. Assume your team consists of 8 developers, and you release once every two months. This results in QA testing a large number of new features all at once in a single release, which increases the likelihood of bugs and lengthens the feedback cycle between the developers and the QA. Expediting the delivery further, and reducing the risk of delivering too many changes all at once, can be mitigated by deploying small changes frequently.
Manual Deployment
Automating the deployment to staging environment, by implementing Continuous Delivery practice, replaces one more manual step, which reduces the operational overhead. If enabled, the system automatically deploys to your staging environment on each pull request merge. This even reduces unnecessary communication between QA and the development team; such as the QA following up with the developers whether they deployed on the staging environment or not, and when.
Once your system has enough code coverage, you’re happy with the QA report, and have a mechanism to quickly rollback from production in case something wrong happens, it is time to move from Continuous Delivery to Continuous Deployment (Full CD), and maybe remove the release testing phase. In fact, by this stage, you’re releasing on each pull request merge, maybe several times a day, which eliminates the need for a staging environment and release testing.
Complex Git Workflow
Fully automated workflow also reduces the operational burden of managing the team’s git repositories. Prior to full automation, a team might need to manage main, develop, releases, features and hotfixes branches, where with full automation, git workflow is much simpler, where it consists of main and feature branches only.
Git Flow — Prior to full automation
Git Flow — Post to full automation
Waste
The best use of a development team’s time is spent on building the right thing for their customers. Having that said, anything else could be waste of time. Fixing bugs, back-and-forth communication, manual operations, or working on features that don’t add value are forms of wasting time. That’s said, it is essential to measure how many items go from the testing phase back to implementation, either due to requirements misunderstanding or reported bugs. Once new features are shipped, monitoring if customers use those features tells whether the team worked on the right features for the customers. Although, this might look like a business decision, the development team should hold themselves accountable to accepting reasonable requirements which add value to their product.
Back-and-forth Communication
Measuring the average movement of a working item between QA and implementation steps, which could be extracted easily on project management tools such as Jira, highlights a problem with the workflow. Maybe the product manager is not writing clear user stories, the QA is not writing detailed acceptance criteria, or there is lack of communication between the business and development members. Regardless of the reason, having such a metric should bring the team’s attention to a problem within their process.
Escaped Defects
Measuring the severity and amount of escaped defects to the production environment helps understand the maturity of the safeguards in place, the quality of the testing phase, and development to production environment's parity. Measuring MTTA (Mean Time to Acknowledge) helps understand how long does it take the development team to beware of production issues and whether they have enough monitoring and alerting tools in place. While measuring MTTR (Mean Time to Recovery) gives an insight about how long it takes to recover from a failure in production, again, highlighting bottlenecks in deployment, scalability and maintainability.
Feature Usage
Building and shipping a feature or system that no one uses is a total waste of time, no matter how fast it was shipped or how solid it is. Development teams should be curious to learn how customers are using the features they shipped. It’s worth asking their business partners about the impact of each requirement, and once shipped, understand whether it met the anticipated impact or not.
Waiting
Obviously, waiting on a step to complete a task will lead to wasting the development team time, such as waiting on code review, manual testing or deployment. Project management tools, such Kanbantool.com or Jira, provides a cumulative flow diagram, which visually shows how much accumulative time each step takes to complete, that can help you quickly detect bottlenecks.
Hosting Cost
Tracking hosting cost regularly, especially when using cloud services, helps teams ensure they avoid unexpected charges, which could result from forgetting to remove a resource that was used for experimentation or unnecessary scaling. Such insight could lead development teams to redesign or refactor part of their systems to lower the cost further.
Putting it all together
To reduce wasted efforts and expedite the delivering value to your customers, start by defining your team metrics, setting goals for each metric, and tracking them regularly. While understanding and improving workflow bottlenecks is essential for delivering value early, it’s equally important to cultivate a healthy team culture and keep your team motivated.
Weekly Tracker: Production Environment Health
- Escaped defects into production
- Mean Time to Acknowledge (MTTA): tracks how quickly the team becomes aware of production issues
- Mean Time to Recovery (MTTR) measures how long it takes to resolve production issues after detection
- Hosting Cost: tracks (cloud) hosting costs to avoid unexpected expenses and identify opportunities for cost optimization
Bi-Weekly Tracker: Team’s Internal Process Health
- Completed Story Points for teams adopting Scrum
- Raised Defects during testing
- Ensure Work In Progress (WIP) Milestones are on track, which should translate to sprint goals for agile teams.
- Average Code Review Time: increased code review time delays shipping features to production and impacts developers productivity by context switching
- Frequency of Releases: deploying small changes frequently should be preferrable over infrequent big depolyments. The latter introduces the risks of breaking an existing system and deployment failure
- Back-and-Forth Communication: measures how often a task moves between QA, development and product manager, indicating inefficiencies in communication and the quality of user stories writing
- Cumulative Flow Diagram: highlights bottlenecks within the development workflow, such as testing and deployment, and sheds light on manual processes to be automated
- Code Coverage: assesses the risk of breaking the existing system on new code check-in, and the readiness for Continuous Deployment
Monthly Tracker
- Feature Usage: monitors how often customers use newly shipped features, helping assess if the development team is building what is right for customers
- User stories in the backlog that are never picked up: might highlight inefficiencies in planning and backlog refinement activities
Software development teams often encounter various bottlenecks that can hinder their ability to deliver value to customers. Bottlenecks include unnecessary workflow steps, delays in code review, manual testing processes, and complex Git workflows. Focusing on automation, especially Continuous Delivery (CD) and ultimately Continuous Deployment (Full CD) can help reduce operational overhead and create simpler workflows. In addition, it’s important to prioritize customer needs over internal processes. Some useful metrics such as code review time, feature usage, and escaped defects can help monitor and improve team performance. By embracing these strategies, teams can minimize wasted efforts and improve their capacity to deliver valuable and reliable features faster.
