The art of transforming testing data into project information

A while back a colleague asked me for help creating a better testing report for her current project. The task got me thinking about how QA managers handle the same information differently; and how this makes the difference between being treated as “Testing Guy” or as the “Inside Information Provider” of a software project.

We are all required to report our work efforts on a regular basis, as a means to communicate project progress status. Many times the test management tools we use produce reports automatically and we just use whatever default reports it spits out and then distribute them to all involved, but that is a big mistake!

I’ve learned that when working with people outside our testing teams we need to think like them, understand what information they need and what format will help them understand it faster and better. More often than not I have found that the default “dry numbers” reports don’t get the proper message across.

Here are 2 simplified examples of different reports:

Example 1. Test execution report
Team A’s Report:
Total Tests in Cycle: 376
Passed Tests: 301
Failed Tests: 28

Tests not Run: 57

Team B’s Report:

Tests in Cycle: 376
Execution percentage: 87.5%
Passed percentage: 80%

It’s all about Framing

In the simple example above, both teams are providing the same data. But who is providing better information? When writing your report remember that people don’t like doing algebra equations in their heads. It is important to understand what information they are looking for, in this case they want to understand (a) how advanced in the testing cycle the team is –> execution %; and (b) what is the status of the application at this point –> passed %.

Example 2. Defects’ report
Team A’s Report:
Total Detected Defects: 453
Closed Defects: 321
Open Defects: 76
Postponed Defects: 56
Total Defects for Release: 397
(Postponed Defects – 56)

Team B’s Report:
Percentage closed: 80.8%
Percentage open: 19.2%
Defect detection rate (2w): 3.2 bugs/day
Defect closure rate (2w): 5.2 bugs/day

Again here, both teams provide the same data but team B also provides more valuable information. Not only did they present the numbers as percentages, but they also point at the convergence status of their project by informing that during the last 2 weeks the fixing rate has surpassed the detection rate.

Tips of the trade

There are many more examples, but the points to remember when defining reports are:

  • The less people need to think, the more intelligent my report will appear to stakeholders; as much as possible I provide my data as percentages or rates.
    PractiTest dashboard item: run status by numbers  PractiTest dashboard item: run status by percent
  • Anticipate the questions and provide the information up-front. If during the status meetings people always ask me for a specific datum, I start including it in my report.
  • Don’t overfill the reports with useless data; too much information will drown the important stuff.
  • Using graphs instead of numbers usually provide 3 to 5 times more information.
    PractiTest dashboard item: issues by status

Finally the most important piece of advice is that a report should talk for itself; the less explanations are needed, the better you are at doing your job.

Creating reports

I would even further recommend creating different visuals & reports to the separate stakeholders in your project, as each audience has different interests regarding project QA. For instance, your managers might care more about time and budget spend on testing and what value that has brought, while HR might care more about team productivity, and your users would care more about when the latest feature update for instance will be released.

Many test management tools today offer a dashboard and/ or reporting features to help with this task. So it’s very easy to execute these reports, and the “hard” part becomes thinking on what data to present , to whom and how to frame it.
For creating a metrics plan I recommend This Ebook

In PractiTest (as I am it’s chief solution architect) we have taken this one step further and created “External dashboards” alongside the usual In App dashboard display. This allows our users to not only easily create, but also share and embed their visual reports with anyone related to their projects, and not just with logged-in team members. You can read more about this feature here.

What tips for presenting data do you use on a daily basis?


 

*Editor’s note: This post has been updated to be more relevant since it was originally posted in December, 2007.

, , , , ,

4 Responses to The art of transforming testing data into project information

  1. Simon Godfrey June 1, 2017 at 1:22 pm #

    Nice post, Joel. When reporting on test progress I try and keep the numbers brief and easily understandable, as you’ve advocated. I also add written information to expand on the failures or provide a visual map of quality so that other stakeholders can understand what the failures are. At this point I’m adding context to the failures, because that’s what people care about right? Are those 20% failures a bunch of bugs we need to worry about, or not? Are we ready to release, or not? So for me a typical report is Numbers, Progress & Explanation of failures (which may be priority, impact to customer and/or impact on the number of failing tests).

  2. QA Guru June 7, 2017 at 1:12 pm #

    Well, here are sharing such a great and helpful information regarding Quality Assurance (QA). Thanks for the publish great ideas in this blog!

Trackbacks/Pingbacks

  1. Convert testing data into valuable project information. – My Blog - October 28, 2016

    […] Read More […]

  2. Testing Bits – 5/28/17 – 6/3/17 | Testing Curator Blog - June 4, 2017

    […] The art of transforming testing data into project information – Joel Montvelisky – http://qablog.practitest.com/the-art-of-transforming-testing-data-into-project-information/ […]

Leave a Reply

Shares