You must choose correctly what NOT to measure!

It’s not only about what you have in your metrics,
it’s also about what you leave outside of them.

This story begins one day, just like any other day…

The whole R&D department of company XYZ Ltd. is sitting in the Open Space area, listening to an interesting presentation about Kanban by one of the Team Leads.  He is explaining how this process is helping his team to achieve faster deliveries and how it provides better visibility and control over the end-to-end process.

The idea of the Company is that starting from the next sprint (Agile sprint, just in case), all the teams will switch from working under Scrum to some sort of Scramban (Scrum + Kanban) that will allow them to focus on a limited number of tasks, in order to develop their product faster and with more visibility to all the company stakeholders.

The last slide of the presentation is about the 5 things that are “very important” to measure, to ensure the team is working under the principle of Kaizen or continuous improvement:

  1. WIP (Work in Progress)
  2. Lead Time
  3. Waste (efficiency)
  4. Throughput
  5. Quality

And here is where the session turned interesting.  When someone in the crowd (someone, for example Me) asked the Team Lead how do they measure Quality in his team?  And then all hell broke loose…


Tell me who I am and I will tell you where to find me

The question may sound simple at first – how do you measure quality? –  but if you ask 10 persons from different teams in your organization to define Quality, you will find between 7 and 15 different definitions (some people will not be able to provide you only “one” answer…).  So, who’s Quality will you choose to measure?

Let’s move further down this road, when I asked the forum of Development Team Leaders, once the session was over and we got together to review this point in more detail, we were able to reach the conclusion that we wanted to measure Quality by taking a look at the bugs (bugs are not the only indication to measure quality, but it is usually the simplest and fastest way to start measuring!).

But then we immediately started arguing about what bugs to measure?  Bugs reported “as part of the development”, bugs detected “once the product was released”, or bugs “detected AND corrected once the product was released”…

In short, everybody has an opinion and all of them are valid in one way or another.


What you measure may improve,
what you don’t measure will always fall behind

The biggest problem with metrics is the psychological effect it has on the people being evaluated.

Start measuring something and you will see how it improves:

  • How far or fast you run.
  • How much food you eat.
  • How many hours a week you spend with your wife and family.
  • How many bugs you find or tests you run.
  • etc.

But since your resources and your capacity are limited, then for every thing you improve there will need to be something else that “pays the price” for it:

  • You will be able run farther or faster because you train a lot, but your grades at school may come down because you don’t have enough time to study for your tests.
  • You will eat less, but you will also smoke more because you are always hungry.
  • You will spend more hours a week with your wife and family (and you will still work the same amount of hours because you don’t want to get fired!), but you will have less friends since you don’t have time to spend with them anymore.
  • You will find more bugs, but more of your bugs will be rejected as “not clear enough”, as “unable to reproduce” or as “duplicates” because you spend less time writing them and reviewing them.
  • You will run more tests, but since you don’t have more time to run them they will be less complex and won’t represent the user’s scenarios good enough, so in the end you end up releasing more bugs to the field.


What’s the solution?  Is it to stop measuring?
Definitely NO!

The solution is not to stop measuring, as I said before measurements are one of the most effective ways or tools you have to improve things.  But you need to be SMART and take into account that it’s not only about what you have in your metrics, it’s also about what you leave outside of them.

For example, if you will start measuring the speed with which you develop each of your features make sure you also measure the quality of your deliverables.  Take also into account that if you develop each feature faster and don’t harm the level of quality, then you will most probably need to develop less features overall – at least at the beginning of the process.

The same goes for every other measure you can think of!  There’s no free lunch, someone or something will always need to eventually pay the bill (or wash the dishes…)


Define an acceptable price you are willing to pay up-front

A good way to go around when you define a “Set of Measurements” for your team is to take into account the following 3 principles:

  1. Keep your metrics simple, easy to calculate, and logical to understand.
  2. Don’t use a single metric.  Define a set of measurements with the intention of covering your process and your needs from 4 or 5 different and complementary angles.  On the other hand don’t set more than 4 or 5 metrics at a single time!
  3. Provide guidance to your team, either by stating this in writing or by making this common knowledge, about the price you are willing to pay in order to improve your measurements.  In plain English: what other aspects of your operation will you be willing to harm in order to achieve improvements on the measured areas.

By doing this you are making sure that all your team is in sync and this will increase your chances of really improving your process, and maybe more importantly of getting the correct feedback needed in order to modify your process (and your measurements) quickly in case they are harmful or counter-productive.


Got any experiences to share?  Feel free!

Maybe you have some interesting stories to share about metrics, please feel free to add them as comments!

From my experience, many of us have “lived through” metrics-abuse-catastrophes in one or more occasions, so I’ll be happy if for a change some of you have been part of experiences or have insights about situations when metrics were used smartly (maybe also in an unconventional way!) and were able to really make a difference…

About PractiTest

Practitest is an end-to-end test management tool, that gives you control of the entire testing process - from manual testing to automated testing and CI.

Designed for testers by testers, PractiTest can be customized to your team's ever-changing needs.

With fast professional and methodological support, you can make the most of your time and release products quickly and successfully to meet your user’s needs.

, ,

7 Responses to You must choose correctly what NOT to measure!

  1. Lalit Bhamare July 5, 2011 at 10:51 am #

    Brilliant post Joel. I was very much in need of something like this.  Yes, I have a story but its mostly a frustrating one.  🙁

    There is this group in my organization who have authority to define metrics for whole organization and they has authority to ask QA people for the compliance to their own definitions of Metrics.  (sadly.. they are not aware what they exactly want to measure).

    Let me share on classic example: ( I personally believe that the way they ask us to measure things.. are leading us to NO WHERE ) . 

    Test Execution Productivity = **No. Normalized Test Cases/ Execution Efforts in Person Days 

    **No. Normalized Test Cases= Sum Total of No. of Steps in all Test Cases/10  

    By following this so called formula sometimes Productivity Comes out to be 40+ and they deny that one person can't execute 40 Test Cases a day. 

    Are these guys kidding me???   

    The reason I shared this is “Am completely annoyed by such mandates” where people who have brought this in, are themselves not aware of what they are measuring and how they are measuring. 

    I appreciate your post for making it clear. I hope.. sometime somehow.. those guys get to read this. 🙂 


  2. joelmonte July 5, 2011 at 5:31 pm #

    Hi Lalit,

    You are not alone!  I specially *like* it when the person bringing these arbitrary and draconian measurements are CONSULTANTS that come with their one-size-fits-all solutions…

    Other than trying to make your management and their management (and so on!) I don't think there is nothing much you can do.  Specially since the scenario you are describing is something that you tend to see most on larger firms where the managers can get (let's say) disconnected to what is really happening in the trenches…

    Good luck!


  3. BrainsLink July 5, 2011 at 1:48 pm #

    Excellent points! There are many project activities that can be measured and there are even more ways in which they can be measured. It's important to begin measuring and recording. The initial metrics can and should be simple to gather and simple to interpret. As the team matures, the metrics should mature also enabling the team to dig deeper and understand more.

    Vague statements like let's “improve quality” or “increase performance” are meaningless. You must be able to measure whatever it is that you want to make better.

  4. joelmonte July 5, 2011 at 5:32 pm #

    Yup, I agree!


  5. joelmonte July 12, 2011 at 11:58 am #

    Thanks for sharing this link by Deming, impressive to see how on 1984 he was able to foresee so many things we now know where absolutely right!

    I think he has a very valid point about making sure we measure also the invisible (or maybe less trivial?) metrics and not only the obvious ones.  In one place we did this by measuring what we called second or third derivate metrics such as customer rejection or acquisition over time, but as Deming said it is something that will be valuable 18 to 24 months from the moment of release and so it was very hard to convince management to be patient and wait for this information.

    In any case, thanks for bringing this point forward and so enrich the discussion!


  6. ElizaF July 19, 2011 at 8:13 am #

    You have hit bang on an issue I have had with metrics for so long. They are generally a set of values arbitrarily inflicted on a SDLC, generally inherited from somewhere else and have little bearing on the project they are being used for.

    It tends to happen when a consultant or a new manager comes into a project bringing with them a solution for measuring quality that worked somewhere else or they have been using for so long that they cannot see the inherent flaws in it.

  7. joelmonte July 19, 2011 at 8:42 am #

    I think I am even guilty of what you wrote about consultants walking in with their metrics and methods…  

    Some years ago I was a young hot-shot thinking I could walk in with “instant-answers” (including metrics!) to companies and situations that were a lot harder to understand that any consultant can solve in a 5 or 10 hour evaluation.

    In any case I learned the hard way that every company, product and team are different, and that there are no magic pills you can hand out.  Only hard work and evaluation from the trenches that will give you the visibility to suggest gradual changes and improvements.

    Glad you liked the post!


Leave a Reply