I know some people have issues with metrics, but in all my years in testing I’ve seen that most (if not all) of these people actually had an issue with the wrong use of metrics and not with the metrics themselves.
After all, just like a knife or fire or even TNT, metrics are nothing more than a tool that can be used for good or for bad.
To slightly modify the old saying:
A fool with a tool is not just a fool,
it is someone who can potentially do a lot of damage too.
(If you disagree with me on this I’d love to hear about it in the comments below this article).
Be careful when using metrics
What can be so dangerous when you work with metrics you ask yourself?
Many many things!
Here are some of the issues that I tend to be careful with when working on my metrics and dashboards:
1. Metrics can be molded to tell the story you want to believe
Some people will look at a number or a chart and see the information that strengthens their own believes.
I remember a project once when 2 managers looked at the exact same graph, and one used it to show how the project was a complete disaster, while the other explained how based on it we were headed on the right path.
When you are providing metrics try to place it in the correct and complete context. Make sure that even if the graph is not self-explanatory (we will touch on that one shortly), it still shows at least a high level direction.
2. Each metrics usually tells the story ONLY from one point of view
A metric tends to show a 1-dimensional or in the best case a 2-dimensional view of a project. When you are trying to paint the complete picture of a complex project you will need to use a number of metrics that show multiple points of view.
This is the best way to provide a complete and accurate picture of your project.
3. Metrics are not always self explanatory
I have heard it many times that a good metric needs to be self explanatory.
I mean, really?
That is nonsense!
When we work on complex projects we cannot expect our metrics to be simple.
In my case, I always make sure to add a short caption or explanation next to each of my graphs so that I make sure people can understand what I see in this metric.
Worst case scenario, if someone does not agree with me we can always review this point of view and I am never afraid to change my point of view.
4. People tend to “fill in the blanks” and jump to conclusions when they see only part of the metrics
Very close to point 2, where we said that you need more than one metric to tell the whole picture, you need to make sure to release and post all your metrics together so that whoever is reviewing them will be able to see them one next to the other.
Human nature is tricky, and we are programmed to “fill in the blanks” when we only have part of the information available. The problem here is that this “filled in info” may not be accurate.
Metrics are a powerful way to express data, project progression, ROI and anything else you want to present, so choose your words wisely.
For more about Testing Metrics I invite you my recent webinar recording, which focused on this exact topic a few weeks back. Any comments are welcome.