Be careful with your Metrics

I am a strong believer in chart

I know some people have issues with metrics, but in all my years in testing I’ve seen that most (if not all) of these people actually had an issue with the wrong use of metrics and not with the metrics themselves.

After all, just like a knife or fire or even TNT, metrics are nothing more than a tool that can be used for good or for bad.

To slightly modify the old saying:

A fool with a tool is not just a fool,
it is someone who can potentially do a lot of damage too.

(If you disagree with me on this I’d love to hear about it in the comments below this article).

Be careful when using metrics

What can be so dangerous when you work with metrics you ask yourself?

Many many things!

Here are some of the issues that I tend to be careful with when working on my metrics and dashboards:

1. Metrics can be molded to tell the story you want to believe

Some people will look at a number or a chart and see the information that strengthens their own believes.

I remember a project once when 2 managers looked at the exact same graph, and one used it to show how the project was a complete disaster, while the other explained how based on it we were headed on the right path.

When you are providing metrics try to place it in the correct and complete context.  Make sure that even if the graph is not self-explanatory (we will touch on that one shortly), it still shows at least a high level direction.

2. Each metrics usually tells the story ONLY from one point of view

A metric tends to show a 1-dimensional or in the best case a 2-dimensional view of a project.  When you are trying to paint the complete picture of a complex project you will need to use a number of metrics that show multiple points of view.

This is the best way to provide a complete and accurate picture of your project.

3. Metrics are not always self explanatory

graphs that don't make senseI have heard it many times that a good metric needs to be self explanatory.

I mean, really?
That is nonsense!

When we work on complex projects we cannot expect our metrics to be simple.

In my case, I always make sure to add a short caption or explanation next to each of my graphs so that I make sure people can understand what I see in this metric.

Worst case scenario, if someone does not agree with me we can always review this point of view and I am never afraid to change my point of view.

4. People tend to “fill in the blanks” and jump to conclusions when they see only part of the metrics

Very close to point 2, where we said that you need more than one metric to tell the whole picture, you need to make sure to release and post all your metrics together so that whoever is reviewing them will be able to see them one next to the other.

Human nature is tricky, and we are programmed to “fill in the blanks” when we only have part of the information available.  The problem here is that this “filled in info” may not be accurate.

In Summary

Metrics are a powerful way to express data, project progression, ROI and anything else you want to present, so choose your words wisely. 

For more about Testing Metrics I invite you my recent webinar recording, which focused on this exact topic a few weeks back. Any comments are welcome.


About PractiTest

Practitest is an end-to-end test management tool, that gives you control of the entire testing process - from manual testing to automated testing and CI.

Designed for testers by testers, PractiTest can be customized to your team's ever-changing needs.

With fast professional and methodological support, you can make the most of your time and release products quickly and successfully to meet your user’s needs.

6 Responses to Be careful with your Metrics

  1. T.J. Maher February 19, 2016 at 10:46 pm #

    It’s like my favorite quote: “Torture numbers, and they will confess to anything!”

    We are having that problem at work… the automated tests are supposed to keep track of failures with our product.

    If the tax service goes down for our test environment, it isn’t a failure of the product… so should you mark it as a pass when you rerun the test again, and it passes?

    If the automated test was broken by an incoming change, but the underlying component that was being tested passed, should you mark the test as passed?

    And how do you track the original results if you mark things as passed?

    Measuring results can be tricky things.

    -T.J. Maher
    Boston, MA

  2. Swati Seela April 20, 2016 at 3:58 pm #

    So true. I am very skeptical about recommending a metric to any team before I am absolutely sure what they are looking for. What works for one, does not work for another- especially when measuring isn’t absolute! Wonderful article.


  1. Five Blogs – 11 February 2016 – 5blogs - February 11, 2016

    […] Be careful with your Metrics Written by: Joel Montvelisky […]

  2. Testing Bits – 2/7/16 – 2/13/16 | Testing Curator Blog - February 14, 2016

    […] Be careful with your Metrics – Joel Montvelisky – […]

  3. Some kick ass blog posts from last week #6 - Mr.Slavchev() - February 15, 2016

    […] Awesome post by Joel Montvelisky on metrics and how to use them. Well, as we all know, “metrics could be always misinterpreted” and this article proves it, one more time: Be careful with your Metrics […]

  4. You need to test less, and think more about the quality of your project. - QA Intelligence - October 21, 2019

    […] you have heard this before, but most probably have not thought it all the way through:  Metrics and measurements are a very dangerous […]

Leave a Reply

As of May 2022, the blog has moved to PractiTest's main website. To keep following our posts go there.