Be careful with your Metrics

I am a strong believer in metrics.bar chart

I know some people have issues with metrics, but in all my years in testing I’ve seen that most (if not all) of these people actually had an issue with the wrong use of metrics and not with the metrics themselves.

After all, just like a knife or fire or even TNT, metrics are nothing more than a tool that can be used for good or for bad.

To slightly modify the old saying:

A fool with a tool is not just a fool,
it is someone who can potentially do a lot of damage too.

(If you disagree with me on this I’d love to hear about it in the comments below this article).

Be careful when using metrics

What can be so dangerous when you work with metrics you ask yourself?

Many many things!

Here are some of the issues that I tend to be careful with when working on my metrics and dashboards:

1. Metrics can be molded to tell the story you want to believe

Some people will look at a number or a chart and see the information that strengthens their own believes.

I remember a project once when 2 managers looked at the exact same graph, and one used it to show how the project was a complete disaster, while the other explained how based on it we were headed on the right path.

When you are providing metrics try to place it in the correct and complete context.  Make sure that even if the graph is not self-explanatory (we will touch on that one shortly), it still shows at least a high level direction.

2. Each metrics usually tells the story ONLY from one point of view

A metric tends to show a 1-dimensional or in the best case a 2-dimensional view of a project.  When you are trying to paint the complete picture of a complex project you will need to use a number of metrics that show multiple points of view.

This is the best way to provide a complete and accurate picture of your project.

3. Metrics are not always self explanatory

graphs that don't make senseI have heard it many times that a good metric needs to be self explanatory.

I mean, really?
That is nonsense!

When we work on complex projects we cannot expect our metrics to be simple.

In my case, I always make sure to add a short caption or explanation next to each of my graphs so that I make sure people can understand what I see in this metric.

Worst case scenario, if someone does not agree with me we can always review this point of view and I am never afraid to change my point of view.

4. People tend to “fill in the blanks” and jump to conclusions when they see only part of the metrics

Very close to point 2, where we said that you need more than one metric to tell the whole picture, you need to make sure to release and post all your metrics together so that whoever is reviewing them will be able to see them one next to the other.

Human nature is tricky, and we are programmed to “fill in the blanks” when we only have part of the information available.  The problem here is that this “filled in info” may not be accurate.

In Summary

Metrics are a powerful way to express data, project progression, ROI and anything else you want to present, so choose your words wisely. 

For more about Testing Metrics I invite you my recent webinar recording, which focused on this exact topic a few weeks back. Any comments are welcome.

 

  • Pingback: Five Blogs – 11 February 2016 – 5blogs()

  • Pingback: Testing Bits – 2/7/16 – 2/13/16 | Testing Curator Blog()

  • Pingback: Some kick ass blog posts from last week #6 - Mr.Slavchev()()

  • It’s like my favorite quote: “Torture numbers, and they will confess to anything!”

    We are having that problem at work… the automated tests are supposed to keep track of failures with our product.

    If the tax service goes down for our test environment, it isn’t a failure of the product… so should you mark it as a pass when you rerun the test again, and it passes?

    If the automated test was broken by an incoming change, but the underlying component that was being tested passed, should you mark the test as passed?

    And how do you track the original results if you mark things as passed?

    Measuring results can be tricky things.

    -T.J. Maher
    Boston, MA

  • Swati Seela

    So true. I am very skeptical about recommending a metric to any team before I am absolutely sure what they are looking for. What works for one, does not work for another- especially when measuring isn’t absolute! Wonderful article.

Shares