Sometimes measuring only one thing
can be worst than not measuring anything at all.
A QA colleague forwarded me a report he sent to his management where he pointed at the fact that the whole R&D Organization was not delivering products at the pace they used to do it in the past.
He was airing out a “known” general concern shared by many testers and developers, and to do this he searched for some “hard data” that would prove this point.
The report was very simple, and it compared the number of releases by all the teams for the last period vs. the period immediately before this one; and it showed that in the last period all the teams together had only delivered about 40% of the releases they did previously.
Simple, right? – WRONG!
What happened with the report?
Immediately after he sent the report, the Team Leads of two of the development teams answered with short emails “explaining” that in the last period they had also had a number of holidays, and attributing everything to this fact.
With this excuse the value the report went down and the subject was erased from Management’s mind and agenda…
The “funny thing” is that my friend had also seen this point, but he thought everyone would see the difference was too large in order to be caused solely because of the holidays!
What can we learn?
There are some basic lessons we can learn from this issue that apply to most organizations
1. Don’t use only one number in order to draw a complete picture
A basic mistake made by many testers and QA Managers is to base all their conclusions on only one number or metric. Basing everything in one measurement is like reaching your conclusions based on only one side of the story instead of collecting information from all possible sides.
So what do you need to do? Look for more than one angle in order to draw a complete picture.
In the case above, the manager could have looked for information about the number of issues detected during the process, or amount of rejections from the field, or at least start by “normalizing” the number of deliveries based on the total number of working days in both periods.
2. Take into account your workplace politics
It is naive to thing that you will send a report pointing at a serious problem and whoever is responsible for it will simply step forward and happily take the responsibility. Victories have thousands of parents, while defeats are usually orphans.
This means that whenever you point to an issue in the system you need to make sure to cover all your angles and to provide information that is, as much as possible, irrefutable.
An important (& ethical) thing to do is to contact the person who may be responsible for the issue and make sure you communicate your findings to him ahead of time, before sending out the report to everyone else. This will give him a chance to come up with ideas on what caused this and even provide as some solutions that you can include in your report.
3. Metrics should include comparable historical information
In the case above, what could have taken down the argument about the holidays?
To compare the number of deliverables not to the period before, but to the same period last year!
Not all metrics are trivial by themselves, and so you may need to compare information in order to see the real differences (or similarities) between them.
In the case of time-based measurements, you need to think about seasonality factors such as:
- Season or time of year
- Part of the quarter
- Time of day
etc.
4. Take into account external factors
As much as you want to be able to measure everything with raw numbers there will always be things you cannot measure that affect your process. You need to research these factors and take them into account in your report.
In the case above, it would have been as simple as stating that in the last period there were also some out of the ordinary holidays, and then add a note saying that even if you factor them into account they would not lower the productivity bellow 80% of the regular level, leaving another 40% decrease in productivity unaccounted for.
External factors will vary all the time, they are the out-of-the-ordinary things that will be trivially visible for people who are living the process. Some examples can be:
- Extraordinary team growth or reduction
- Extreme product shifts or market disruptions
- Things like prolonged strikes
- And even in some cases, as I saw not too long ago in a company, unusual number of births in a team (5 out of 8 team members received a child in a period of 3 months)
Bottom line
One of the main difference between a credible story and plain gossip is that a story has more than angle to it. Your metric’s-based-report is basically a way to tell a story of what is happening in your product/process/team.
So, in your next report or fact-based email make sure you take into account:
1. To have more than one metric or angle measured
2. Company politics
3. Historical information
and
4. Extraordinary factors
