Podcast – Metrics – The good, the bad and the ugly

How’s life


  • Back from some the QA or the Highway Conference in Ohio, visited some customers in the US and Canada (literally buried in snow).
  • Working on the State of Testing


  • Exciting news, got my own office and properly going knee-deep into getting my online management course completed and launching my Parent brand Parent Brain.

Is there a more controversial topic in testing?  Yet a tool more widely used by testing teams?

We count everything – how many tests have we run, how many bugs, what is the relationship between developers and testers, how many minutes before lunch…

We do this because we are used to it, because our culture is built on comparing stuff, and because when used correctly and in conjunction with other information tools, they metrics can help us to provide some pretty valuable information to the rest of the team and stakeholders.

But it is also correct that many many teams, maybe most teams out there, do not pay the required attention to their metrics.  And in the best of cases they are showing incomplete data, and in the worst of cases are showing data that will steer the boat in the wrong direction altogether.

Let’s talk about metrics, what do you say?

Why are we measuring? What problem are we trying to solve?

  • The majority of measures you hear about don’t appear to be addressing a problem
  • Reality changes and metrics should change accordingly

What gets measured, gets managed

  • Be careful what you measure
  • People will work to the measures
  • The law of conservation of metrics:
    Everything that gets improved due to a metric,
    will come at the expense of something you are not measuring.
  • One-dimensional metrics are dangerous and ultimately bad – need to think on 3 or 4 dimensional metrics that complement each other and give a full picture.
  • What are you not measuring? And why?
  • Who decided on the measure and why?
  • How did they come up with the measure / what problem were they trying to solve and do you still have that problem?
  • Measurements should be used to improve the process, system and world of work – are the measures you’re using doing that?
  • Behaviors are the best measure of success / are you getting better?
  • Gaming measures is common
  • Danger of metrics is the pesticide paradox
  • Never measure people!
  • Metrics should always be public, and explained
  • You need to change, refresh, shuffle your metrics

Guide for Metric definition – S.M.A.R.T.

  • Simple
  • Measurable – numeric
  • Actionable
  • Repeatable
  • Timely

Trends are important

  • A single measure is unlikely to tell you much 
  • Trends are important – over time
  • The variance between the high and low is important / is it consistent and stable, or wildly unpredictable 
  • Measures should be both qualitative and quantitative
No comments yet.

Leave a Reply