It’s not only about what you have in your metrics,
it’s also about what you leave outside of them.
This story begins one day, just like any other day…
The whole R&D department of company XYZ Ltd. is sitting in the Open Space area, listening to an interesting presentation about Kanban by one of the Team Leads. He is explaining how this process is helping his team to achieve faster deliveries and how it provides better visibility and control over the end-to-end process.
The idea of the Company is that starting from the next sprint (Agile sprint, just in case), all the teams will switch from working under Scrum to some sort of Scramban (Scrum + Kanban) that will allow them to focus on a limited number of tasks, in order to develop their product faster and with more visibility to all the company stakeholders.
The last slide of the presentation is about the 5 things that are “very important” to measure, to ensure the team is working under the principle of Kaizen or continuous improvement:
- WIP (Work in Progress)
- Lead Time
- Waste (efficiency)
- Throughput
- Quality
And here is where the session turned interesting. When someone in the crowd (someone, for example Me) asked the Team Lead how do they measure Quality in his team? And then all hell broke loose…
Tell me who I am and I will tell you where to find me
The question may sound simple at first – how do you measure quality? - but if you ask 10 persons from different teams in your organization to define Quality, you will find between 7 and 15 different definitions (some people will not be able to provide you only “one” answer…). So, who’s Quality will you choose to measure?
Let’s move further down this road, when I asked the forum of Development Team Leaders, once the session was over and we got together to review this point in more detail, we were able to reach the conclusion that we wanted to measure Quality by taking a look at the bugs (bugs are not the only indication to measure quality, but it is usually the simplest and fastest way to start measuring!).
But then we immediately started arguing about what bugs to measure? Bugs reported “as part of the development”, bugs detected “once the product was released”, or bugs “detected AND corrected once the product was released”…
In short, everybody has an opinion and all of them are valid in one way or another.
What you measure may improve,
what you don’t measure will always fall behind
The biggest problem with metrics is the psychological effect it has on the people being evaluated.
Start measuring something and you will see how it improves:
- How far or fast you run.
- How much food you eat.
- How many hours a week you spend with your wife and family.
- How many bugs you find or tests you run.
- etc.
But since your resources and your capacity are limited, then for every thing you improve there will need to be something else that “pays the price” for it:
- You will be able run farther or faster because you train a lot, but your grades at school may come down because you don’t have enough time to study for your tests.
- You will eat less, but you will also smoke more because you are always hungry.
- You will spend more hours a week with your wife and family (and you will still work the same amount of hours because you don’t want to get fired!), but you will have less friends since you don’t have time to spend with them anymore.
- You will find more bugs, but more of your bugs will be rejected as “not clear enough”, as “unable to reproduce” or as “duplicates” because you spend less time writing them and reviewing them.
- You will run more tests, but since you don’t have more time to run them they will be less complex and won’t represent the user’s scenarios good enough, so in the end you end up releasing more bugs to the field.
What’s the solution? Is it to stop measuring?
Definitely NO!
The solution is not to stop measuring, as I said before measurements are one of the most effective ways or tools you have to improve things. But you need to be SMART and take into account that it’s not only about what you have in your metrics, it’s also about what you leave outside of them.
For example, if you will start measuring the speed with which you develop each of your features make sure you also measure the quality of your deliverables. Take also into account that if you develop each feature faster and don’t harm the level of quality, then you will most probably need to develop less features overall – at least at the beginning of the process.
The same goes for every other measure you can think of! There’s no free lunch, someone or something will always need to eventually pay the bill (or wash the dishes…)
Define an acceptable price you are willing to pay up-front
A good way to go around when you define a “Set of Measurements” for your team is to take into account the following 3 principles:
- Keep your metrics simple, easy to calculate, and logical to understand.
- Don’t use a single metric. Define a set of measurements with the intention of covering your process and your needs from 4 or 5 different and complementary angles. On the other hand don’t set more than 4 or 5 metrics at a single time!
- Provide guidance to your team, either by stating this in writing or by making this common knowledge, about the price you are willing to pay in order to improve your measurements. In plain English: what other aspects of your operation will you be willing to harm in order to achieve improvements on the measured areas.
By doing this you are making sure that all your team is in sync and this will increase your chances of really improving your process, and maybe more importantly of getting the correct feedback needed in order to modify your process (and your measurements) quickly in case they are harmful or counter-productive.
Got any experiences to share? Feel free!
Maybe you have some interesting stories to share about metrics, please feel free to add them as comments!
From my experience, many of us have “lived through” metrics-abuse-catastrophes in one or more occasions, so I’ll be happy if for a change some of you have been part of experiences or have insights about situations when metrics were used smartly (maybe also in an unconventional way!) and were able to really make a difference…
