A while back a colleague asked me for help creating a better testing report for her current project. The task got me thinking about how QA managers handle the same information differently; and how this makes the difference between being treated as “Testing Guy” or as the “Inside Information Provider” of a software project.
We are all required to report our work efforts on a regular basis, as a means to communicate project progress status. Many times the test management tools we use produce reports automatically and we just use whatever default reports it spits out and then distribute them to all involved, but that is a big mistake!
I’ve learned that when working with people outside our testing teams we need to think like them, understand what information they need and what format will help them understand it faster and better. More often than not I have found that the default “dry numbers” reports don’t get the proper message across.
Here are 2 simplified examples of different reports:
Example 1. Test execution report
Team A’s Report:
Total Tests in Cycle: 376
Passed Tests: 301
Failed Tests: 28
Tests not Run: 57
Team B’s Report:
Tests in Cycle: 376
Execution percentage: 87.5%
Passed percentage: 80%
It’s all about Framing
In the simple example above, both teams are providing the same data. But who is providing better information? When writing your report remember that people don’t like doing algebra equations in their heads. It is important to understand what information they are looking for, in this case they want to understand (a) how advanced in the testing cycle the team is –> execution %; and (b) what is the status of the application at this point –> passed %.
Example 2. Defects’ report
Team A’s Report:
Total Detected Defects: 453
Closed Defects: 321
Open Defects: 76
Postponed Defects: 56
Total Defects for Release: 397
(Postponed Defects – 56)
Team B’s Report:
Percentage closed: 80.8%
Percentage open: 19.2%
Defect detection rate (2w): 3.2 bugs/day
Defect closure rate (2w): 5.2 bugs/day
Again here, both teams provide the same data but team B also provides more valuable information. Not only did they present the numbers as percentages, but they also point at the convergence status of their project by informing that during the last 2 weeks the fixing rate has surpassed the detection rate.
Tips of the trade
There are many more examples, but the points to remember when defining reports are:
- The less people need to think, the more intelligent my report will appear to stakeholders; as much as possible I provide my data as percentages or rates.
- Anticipate the questions and provide the information up-front. If during the status meetings people always ask me for a specific datum, I start including it in my report.
- Don’t overfill the reports with useless data; too much information will drown the important stuff.
- Using graphs instead of numbers usually provide 3 to 5 times more information.
Finally the most important piece of advice is that a report should talk for itself; the less explanations are needed, the better you are at doing your job.
I would even further recommend creating different visuals & reports to the separate stakeholders in your project, as each audience has different interests regarding project QA. For instance, your managers might care more about time and budget spend on testing and what value that has brought, while HR might care more about team productivity, and your users would care more about when the latest feature update for instance will be released.
Many test management tools today offer a dashboard and/ or reporting features to help with this task. So it’s very easy to execute these reports, and the “hard” part becomes thinking on what data to present , to whom and how to frame it.
For creating a metrics plan I recommend This Ebook
In PractiTest (as I am it’s chief solution architect) we have taken this one step further and created “External dashboards” alongside the usual In App dashboard display. This allows our users to not only easily create, but also share and embed their visual reports with anyone related to their projects, and not just with logged-in team members. You can read more about this feature here.
What tips for presenting data do you use on a daily basis?
*Editor’s note: This post has been updated to be more relevant since it was originally posted in December, 2007.