One of the expectations of implementing Scrum is the ability to create potentially releasable increments in every iteration. This implies testing User Stories in the same Sprint as they are being developed. Scrum has changed our approach not only in development but testing also – the process has become even more challenging. These are some issues typically experienced by team members practicing Scrum:

  • Acceptance Criteria is ill-defined or not even existing for each User Story,
  • User Stories are too big and not well defined for a developer and a tester to go over develop-test-fix-test cycle before the sprint times out,
  • lack of knowledge and communication between team members,
  • infrequent deployments create delays,
  • bad processes and tooling.

If you are not producing versioned software increments in each Sprint iteration, then the problem is even worse, as you can not even test against a tagged version of provisioned code.

As mentioned – the problem is also in the tools and/or processes. If you can not detect and count defects, then you have a long way to go in mastering the delivery of quality software. One of the metrics that we use to measure team’s ability to deliver releasable increments is Defect Leakage KPI.

First we have to understand the context of Defect Leakage. In Scrum the team forecasts the delivery of a potentially releasable valuable features in each iteration. That implies (near) bug-free code deployed to a presentable runtime environment. Preferably, most of the bugs should be caught in the same sprint as the User Story is being developed in. The number of detected defects after iteration should be as small as possible. Defect Leakage KPI shows relationship between the number of detected defects after iteration and the number of detected defects during iteration. The image below represents the chart for Defect Leakage where each bar represents the ratio of detected defects after sprint with defects detected during the sprint. Ideally that ratio is zero.

Next we’ll see how to capture data for Defect Leakage KPI in JIRA. As already explained, we need two pieces of information for each sprint:

  • number of defects detected during sprint and
  • number of defects detected after the sprint.

One of the approaches to distinguish those two kinds of defects is using ‘Labels’ field in JIRA to mark a defect for appropriate group. For example – you can choose two labels named after-sprint and during-sprint.

This is not the only way to classify defects. For example, you could configure JIRA to use custom fields with predefined vales. When we have defects marked as it was described above, we can execute JQL queries to get the data for Defect Leakage chart.

PROJECT = "AT" AND SPRINT = "AT09" AND ISSUETYPE = BUG AND LABELS = DURING-SPRINT

Assuming the two JQL queries return:

  • 8 for number of defects captured during sprint
  • 3 for number of defects captured after sprint

The mechanics for providing data sets to feed the chart is by manually editing the two input fields for this chart. The other is by using Agile Tools REST API.

The effort needed to provide this data must absolutely be minimal. With a pragmatic approach and a little discipline teams can get an insight into their ability to deliver potentially releasable increments.

The Defect Leakage is a measure in the Ability to Innovate area, which is one of the four Key Value Areas we will cover in Agile Tools. Only in the context of other metrics and with proper interpretation the numbers will tell the story – as they will – in our tool.

The real value and your takeaway is that by observing Defect Leakage you will be changing the way you develop software. You will have to master how to properly write and size User Stories and how to work in short develop-test-fix-test cycles. The benefit is to break silos, to develop more full-stack competences, to have more short conversations between the developer and a (dedicated) tester and most importantly – to deliver value to your customers.