Can Metrics be EVIL? No, but the people who misuse them can…

There have been multiple debates in places such as QAForums about whether Process and Project Metrics can do more harm than good to the Organization.

My view is that Metrics are primarily good; and most problems come from the way we analyze and display the information.
We also tend to forget that metrics are only an Information Channel and not a product by themselves.

evil metrics
Following are some Rules of Thumb I use when creating a metrics system for an Organization:

(1) Remember that metrics will always be a base for comparisons.
If you know that people will compare the metrics you provide, start by providing the base for comparison yourself.
In some companies this means to provide present numbers vs. past numbers and even vs. target numbers; in other companies this also means to provide information for other teams in the Organization.
Just don’t leave this up to the reader himself…

(2) Metrics need to be balanced.
You cannot measure and show only side of process. Make sure that you display information for all or most activities taking place in the team.
For example, you cannot provide only the number of opened bugs, but you need also to show numbers for Run Tests, Written Tests, and any other activities that measure the activities of the Testing Team.

(3) Metrics need to be normalized.
Make sure you are not trying to balance the outputs of a 15-tester team with that of a 3-tester team.

(4) Metrics need to be comparable.
Following normalization, you need to make sure you are not comparing Apples & Oranges.
Don’t try to compare a team working on an established Enterprise Application with that of a Start-Up team working on a Prototype for a Customer Oriented product.

(5) Don’t make Metrics personal.
Don’t make it a witch-hunt for a single person or a single team to take the blame.
NEVER display metrics for a single individual.
You also need to show the information in a way that the Stakeholders see the truth being displayed and not a way of trying to put blame on subjects.

(6) Think of what you want to communicate by publishing your metrics.
Don’t provide dry numbers only, specially not when you are showing possible problems that need to be corrected.
If you know that stakeholders will immediately ask questions such as “how did this happen?” or “what are we doing to correct this?”, make sure such questions are immediately answered in your report with corrective actions from the team.

(7) Metrics evolve.
It is legitimate and even necessary to review what you are measuring once in a while to make sure you are answering the needs of your organization; if the company’s priorities change from time to time so should the parameters being measured.

Most importantly, metrics are only a COMMUNICATION CHANNEL and not an objective by themselves.
They should provide correct and complete information from one group of Stakeholders to the other in order to allow for the correct decisions to be made.

Remember that you need to serve both sides of the communication process.
The best way of doing this is by not taking any sides, and by making sure both sides know this…

About PractiTest

Practitest is an end-to-end test management tool, that gives you control of the entire testing process - from manual testing to automated testing and CI.

Designed for testers by testers, PractiTest can be customized to your team's ever-changing needs.

With fast professional and methodological support, you can make the most of your time and release products quickly and successfully to meet your user’s needs.

4 Responses to Can Metrics be EVIL? No, but the people who misuse them can…

  1. Rob Lambert February 27, 2009 at 6:04 pm #

    Hi Joel,

    Good post. Nice to see a balanced view on metrics, rather than the often stated metrics are bad.

    I’m a firm believer that metrics should only be produced when they are needed and not just for the sake of it. I also strongly believe that Audience and Purpose are the most important factors to consider. I’m working on a post for my communication blog on metrics and will be sure to cross reference your excellent blog post..


  2. Michael February 27, 2009 at 7:34 pm #

    Hi, Joel…

    I invite you (and Rob) to go to

    and read the Kaner and Bond paper that’s linked from it. Doug Hoffman’s paper is important too.

    —Michael B.

  3. Rob Lambert February 27, 2009 at 8:27 pm #

    Hi Michael,

    I’ve read quite a few of these articles and a couple of the books – need to read more for sure and I’m generally a believer that metrics are pretty woeful and not that valuable but at the end of the day people sometimes still request them and I reckon Joels post is a good way of handling the metrics produced when needed. I’ve experienced one extreme to the other with metrics and am generally under the consensus that the metrics I provided offered no extra information than a simple verbal description could have, however, management used these metrics for assessing performance and project quality….

    The Dark Side of metrics is an important paper for me and is something I tend to re-read regularly. I’ve recently got to the point of producing no metrics at all but instead providing simple low tech dashboards and it works perfectly but I know people out there still need to use metrics.

    I’ve just ordered the other books I’d not read before. I’m learning something new every day.


  4. Joel Montvelisky February 27, 2009 at 10:47 pm #

    Rob & Michael,

    I think we've seen the dangers (and the benefits) of software metrics.
    My post comes to provide a "contra" to all the people who categorically and emphatically say that they refuse to measure (or even if they measure they keep their metrics private and hidden).

    I believe in measuring and using metrics carefully to improve our processes, and also our products but on lesser account.

    Rob, as you said many people paint metrics as either all white or all black, and I think we only harm the starting testers by depicting them this way. I am already interested in reading the blog you are working on!

    Michael, thanks for the pointers, I have not read these papers but I just downloaded them and will start tonight.


Leave a Reply