One metric = One Big Mistake

Testing Metrics

Sometimes measuring only one thing can be worse than not measuring anything at all.

A QA colleague forwarded me a report he sent to his management where he pointed at the fact that the whole R&D Organization was not delivering products at the pace they used to do it in the past.

He was airing a “known” general concern shared by many testers and developers, and to do so he searched for some “hard data” that would prove this point.

The report was very simple, and it compared the number of releases by all the teams for the last period vs. the period immediately before this one; and it showed that in the last period all the teams together had only delivered about 40% of the releases they did previously.

Simple, right? – WRONG!

What happened with the report?

Immediately after he sent the report, the Team Leads of two of the development teams answered with short emails “explaining” that in the most recent period there werw a number of holidays, and attributed everything to that fact.

Testing Metrics With this excuse the value the report went down and the subject was erased from Management’s mind and agenda…

The “funny” thing is that my friend had also seen this point, but he thought everyone would see that the difference in productivity was too large to be caused solely by the holidays!

What can we learn?

There are some basic lessons we can learn from this issue that apply to most organizations:

1. Don’t use only one number in order to draw a complete picture

A basic mistake made by many testers and QA Managers is to base all their conclusions on only one number or metric. Basing everything on one measurement is like reaching your conclusions based on only one side of the story instead of collecting information from all possible sides.

So what do you need to do? Look for more than one angle in order to draw a complete picture.

In the case above, the manager could have looked for information about the number of issues detected during the process, or amount of rejections from the field, or at least started by “normalizing” the number of deliveries based on the total number of working days in both periods.

2. Take your workplace politics into account

It is naive to think that you would send a report pointing to a serious problem and whoever is responsible for it will simply step forward and happily take the blame. Victories have thousands of parents, while defeats are usually orphans.

This means that whenever you point to an issue in the system you need to make sure to cover all your angles and to provide information that is, as much as possible, irrefutable.

An important (and ethical) thing to do is to contact the person who may be responsible for the issue and make sure you communicate your findings to him ahead of time, before sending out the report to everyone else. This will give him a chance to come up with ideas about what caused this and even provide as some solutions that you can include in your report.

3. Metrics should include comparable historical information

In the case above, what could have shot down the argument about the holidays?
Comparing the number of deliverables not to the period before, but to the same period last year!

The meaning of metrics is not always obvious in and of itself so you may need to compare information in order to see the real differences (or similarities) between them.

In the case of time-based measurements, you need to think about seasonality factors such as:
– Season or time of year
– Part of the quarter
– Time of day, etc.

4. Take into account external factors

As much as you want to be able to measure everything with raw numbers there will always be things that affect your process that you cannot measure. You need to research these factors and take them into account in your report.

In the case above, it would have been as simple as stating that in the most recent period there was an unusual number of holidays, and then add a note saying that even when you factor them in they should only account for about 40% of the decrease in productivity, leaving the other 60% due to other non-holiday-related factors.

External factors will vary all the time, they are the out-of-the-ordinary things that  are easily visible to people who are living the process. Some examples can be:

– Extraordinary team growth or reduction
– Extreme product shifts or market disruptions
– Things like prolonged strikes
– And even in some cases, as I saw not too long ago in a company, unusual number of births in a team (5 out of 8 team members had a new baby within a 3-month period!)

Bottom line

One of the main differences between a credible story and plain gossip is that a story has more than one angle to it. Your metrics-based report is basically a way to tell a story about what is happening in your product/process/team.

So, in your next report or fact-based email make sure you take into account:
Testing Metrics 1. More than one metric or angle is measured
2. Company politics
3. Historical information
4. Unusual factors

(*picture by digitalart)

About PractiTest

Practitest is an end-to-end test management tool, that gives you control of the entire testing process - from manual testing to automated testing and CI.

Designed for testers by testers, PractiTest can be customized to your team's ever-changing needs.

With fast professional and methodological support, you can make the most of your time and release products quickly and successfully to meet your user’s needs.

, , ,

5 Responses to One metric = One Big Mistake

  1. Joe Strazzere November 14, 2011 at 3:29 pm #

    “all the teams together had only delivered about 40% of the releases they did previously”

    The numbers say what happened, but not why.

    Even normalizing the numbers based on the number of work days in the period would not say why something happened, and would not say if this cause for alarm, or celebration, or something in between. (Perhaps fewer releases were needed during this period compared to last.  Perhaps folks were busy helping out other groups.)

    For expressing the “why”, use words!

  2. Anonymous November 14, 2011 at 4:46 pm #

    Hi Joe,

    I could not agree more with you that pointing at the issues is *ONLY* the first step, and you should always strive to understand the real history behind any set of raw numbers.

    Perhaps I should take on this point in a future post…


  3. Expat Canuck November 20, 2011 at 6:08 pm #

    This is a very insightful article.  It should be noted here that most technical people find company politics to be revolting. 

  4. Anonymous November 27, 2011 at 11:15 am #

    As revolting as I find company politics, it is naive to think we can disregard them and still maintain out level of influence in the company and process.

    My approach is to “play the game” but make sure to keeps my hands clean at all time.

  5. Testing Tools Journal November 28, 2011 at 11:33 am #

    Great post Joel. 

    I guess realizing that we have issues delivering in itself is a big milestone to meet let alone answering “Why”. The one that comes before answering “Why” as i see it in my current work place is “Accepting” that we have issues in delivery and adapting to mindset of doing something about it with a clean soul. 

Leave a Reply