Sometimes measuring only one thing can be worse than not measuring anything at all.
A QA colleague forwarded me a report he sent to his management where he pointed at the fact that the whole R&D Organization was not delivering products at the pace they used to do it in the past.
He was airing a “known” general concern shared by many testers and developers, and to do so he searched for some “hard data” that would prove this point.
The report was very simple, and it compared the number of releases by all the teams for the last period vs. the period immediately before this one; and it showed that in the last period all the teams together had only delivered about 40% of the releases they did previously.
Simple, right? – WRONG!
What happened with the report?
Immediately after he sent the report, the Team Leads of two of the development teams answered with short emails “explaining” that in the most recent period there werw a number of holidays, and attributed everything to that fact.
The “funny” thing is that my friend had also seen this point, but he thought everyone would see that the difference in productivity was too large to be caused solely by the holidays!
What can we learn?
There are some basic lessons we can learn from this issue that apply to most organizations:
1. Don’t use only one number in order to draw a complete picture
A basic mistake made by many testers and QA Managers is to base all their conclusions on only one number or metric. Basing everything on one measurement is like reaching your conclusions based on only one side of the story instead of collecting information from all possible sides.
So what do you need to do? Look for more than one angle in order to draw a complete picture.
In the case above, the manager could have looked for information about the number of issues detected during the process, or amount of rejections from the field, or at least started by “normalizing” the number of deliveries based on the total number of working days in both periods.
2. Take your workplace politics into account
It is naive to think that you would send a report pointing to a serious problem and whoever is responsible for it will simply step forward and happily take the blame. Victories have thousands of parents, while defeats are usually orphans.
This means that whenever you point to an issue in the system you need to make sure to cover all your angles and to provide information that is, as much as possible, irrefutable.
An important (and ethical) thing to do is to contact the person who may be responsible for the issue and make sure you communicate your findings to him ahead of time, before sending out the report to everyone else. This will give him a chance to come up with ideas about what caused this and even provide as some solutions that you can include in your report.
3. Metrics should include comparable historical information
In the case above, what could have shot down the argument about the holidays?
Comparing the number of deliverables not to the period before, but to the same period last year!
The meaning of metrics is not always obvious in and of itself so you may need to compare information in order to see the real differences (or similarities) between them.
In the case of time-based measurements, you need to think about seasonality factors such as:
– Season or time of year
– Part of the quarter
– Time of day, etc.
4. Take into account external factors
As much as you want to be able to measure everything with raw numbers there will always be things that affect your process that you cannot measure. You need to research these factors and take them into account in your report.
In the case above, it would have been as simple as stating that in the most recent period there was an unusual number of holidays, and then add a note saying that even when you factor them in they should only account for about 40% of the decrease in productivity, leaving the other 60% due to other non-holiday-related factors.
External factors will vary all the time, they are the out-of-the-ordinary things that are easily visible to people who are living the process. Some examples can be:
– Extraordinary team growth or reduction
– Extreme product shifts or market disruptions
– Things like prolonged strikes
– And even in some cases, as I saw not too long ago in a company, unusual number of births in a team (5 out of 8 team members had a new baby within a 3-month period!)
One of the main differences between a credible story and plain gossip is that a story has more than one angle to it. Your metrics-based report is basically a way to tell a story about what is happening in your product/process/team.