PractiTest QA Blog

Last night I was invited to give a presentation on the subject of Bugs as part of a SIGiST mini-track.

It’s not uncommon to enjoy a good presentation from the audience perspective.
I mean, you can find the subject of the presentation interesting and new, you can enjoy the way the presenter conveys the message, you can even relate to the person standing in front of the room and feel good only because it could be you instead of him conveying the same “intelligent message” to the audience.

But yesterday I had one of those special occasions when you feel good with the presentation and its dynamic from the Presenter perspective.  And don’t get me wrong, it is not that I have not had good presentations before, but yesterday it felt something slightly better; it was one of those sessions when you feel that you didn’t provide a presentation so much as you were able to facilitate a group discussion around a specific topic.

The subject was not new or particularly exhilarating, what more can you say or discuss about Bugs in Software Testing?  But mainly because of this reason and due to the dynamic I tried to carry throughout the session I think we managed to take this subject and instead of reviewing it from 20,000 feet in 45 minutes, we examined specific points of it with microscopic precision and based on the points of view and experience from some pretty sharp people (some of whom I knew previously, but other who I didn’t and was glad to meet for the first time).

So I wanted to thank Alon and guys from SIGiST and Sela for organizing the event; but I also wanted to thank the people who took part of it and made it a fun and learning experience for all of us.  Gil, Gaby, Tal, Yoav, Yaron, Avi, and the rest of the participants who I am obviously forgetting… thanks for the time and the experience.

Also, as promised, I am posting the presentation in the QA Resources page of our site in case you want to take a look at it.

A PractiTest customer was telling me about a problem they have with people not writing down their reasons when rejecting a bug, or writing unclear explanations in the Description or Comments fields.

I know exactly what she meant…  I also hate this issue plus the fact that since the data is in Free Text it is very difficult to track the reasons and keep statistics (in case you believe like I do in tracking this behaviour as part of an on-going improvement policy).

My solution is simple: I create a customized field called Reject Reason with the acceptable values for rejecting bugs in the project.  Usually these values are: Not a Bug, Not Clear, Unable to Reproduce, and Duplicate.

I find that by providing a list instead of asking my users to write down their explanation in free text they are more willing to comply.

Also since I have this as a structured list, PractiTest provides me automatically with statistics to keep track of my rejections and make corrections if needed.

There have been multiple debates in places such as QAForums about whether Process and Project Metrics can do more harm than good to the Organization.

My view is that Metrics are primarily good; and most problems come from the way we analyze and display the information.
We also tend to forget that metrics are only an Information Channel and not a product by themselves.

Following are some Rules of Thumb I use when creating a metrics system for an Organization:

(1) Remember that metrics will always be a base for comparisons.
If you know that people will compare the metrics you provide, start by providing the base for comparison yourself.
In some companies this means to provide present numbers vs. past numbers and even vs. target numbers; in other companies this also means to provide information for other teams in the Organization.
Just don’t leave this up to the reader himself…

(2) Metrics need to be balanced.
You cannot measure and show only side of process. Make sure that you display information for all or most activities taking place in the team.
For example, you cannot provide only the number of opened bugs, but you need also to show numbers for Run Tests, Written Tests, and any other activities that measure the activities of the Testing Team.

(3) Metrics need to be normalized.
Make sure you are not trying to balance the outputs of a 15-tester team with that of a 3-tester team.

(4) Metrics need to be comparable.
Following normalization, you need to make sure you are not comparing Apples & Oranges.
Don’t try to compare a team working on an established Enterprise Application with that of a Start-Up team working on a Prototype for a Customer Oriented product.

(5) Don’t make Metrics personal.
Don’t make it a witch-hunt for a single person or a single team to take the blame.
NEVER display metrics for a single individual.
You also need to show the information in a way that the Stakeholders see the truth being displayed and not a way of trying to put blame on subjects.

(6) Think of what you want to communicate by publishing your metrics.
Don’t provide dry numbers only, specially not when you are showing possible problems that need to be corrected.
If you know that stakeholders will immediately ask questions such as “how did this happen?” or “what are we doing to correct this?”, make sure such questions are immediately answered in your report with corrective actions from the team.

(7) Metrics evolve.
It is legitimate and even necessary to review what you are measuring once in a while to make sure you are answering the needs of your organization; if the company’s priorities change from time to time so should the parameters being measured.

Most importantly, metrics are only a COMMUNICATION CHANNEL and not an objective by themselves.
They should provide correct and complete information from one group of Stakeholders to the other in order to allow for the correct decisions to be made.

Remember that you need to serve both sides of the communication process.
The best way of doing this is by not taking any sides, and by making sure both sides know this…

On-going improvements are usually sitting quietly under our noses…Taking a closer look at our bugs and the information available in them we can come up with very strong upgrades to our testing process.

Following are 4 metrics you can collect easily during your regular bug lifecycle that can provide good leads into places where you can greatly improve the quality of your testing process:

I. Defect Density (by product area, testing phase, development team, etc)
Goal: Identify risk areas where you need to do more testing.
How to track it? Use one or more list fields to define the product area or module for each detected bug.
Possible Improvement(s): Analyzing a concrete statistical view of the risk areas in the product you can decide whether to modify your testing assignments, schedules or plans to better cope with these risks.
Notes:
(1) Make sure you are not pointing a flash-light at a bonfire calling for the fire department, places with a lot of development work are likely to generate a high number of bugs.
(2) This metric can be used within the process to modify your current project, or during the project post-mortem to provide feedback for future projects.

II. Defect Rejected Reasons
Goal: Understand why your testers are wasting time reporting the wrong bugs, rewriting their bugs better, or arguing with developers unnecessarily.
How to track it? Create a list field with the possible reasons for rejecting bugs, make this field mandatory whenever an issue is set to status rejected.
Rejected reasons vary between organizations, but the list will always include values such as: “Missing Information”, “Duplicate Bug”, “Not a Bug” and “Unable to Reproduce”.
Possible improvements: If the team has a high rate of rejections look for the main problematic reasons and implement corrective actions. These actions may range from creating a new bug description template, to training engineers on how to search for duplicates bugs in your existing repository, or any other action to treat the source of the problem.
Notes:
(1) Sometimes you will need to look for specific team members.
(2) Make sure you have a problem before implementing a cure; it is natural to have rejections, just make sure they are keep at acceptable levels.

III. Escaping Defect Reason
Goal: Map problematic areas where defects are been constantly reported by customers.
How to track it? Whenever a bug is reported from the field register whether it was previously detected by the testing team or not . If it was detected why was it left unfixed, and if it wasn’t found explain why.
Create a list field to register this information. Possible values for this list are: “Found – Low Priority”, “Found – High Risk”, “Missing Test Scenario”, “Border-case Scenario”, “Missing Environment”, “Border-case Environment”, etc.
Possible improvements:
(a) Increase your testing scenarios and/or environments with information from the field.
(b) Study application usage patterns in order to become a better “customer advocate” and provide better priority and severity information to your tests and bugs.
Notes: It is unrealistic to detect and fix all the defects, but we need to make sure we are not missing some of those we would really like to find in our process.

IV. Corrective Test Actions
Goal: Correct cases where bugs were caught by chance or to late in the process in order to be fixed without incurring in delays or risks.
How to track it? Create a Memo Field called “Corrective Test Actions” where testers or developers fill information that should be considered in order to create new test cases or modify existing ones.
Whenever a bug is detected by chance or too late in the process the responsible tester or developer should provide suggestions or comments indicating what needs to be corrected in the process or test scenarios.
Possible Improvements: At the end of the cycle or project all the bugs that have data on the Corrective Test Actions field should be examined and their comments be submitted as part of the feedback to the Test Manager and to the general post-mortem process
Note: There can be some overlap between the this metric and the previous one (Escaping Defect Reason).

A colleague once observed that “not all bugs are created equal…” and thus you can choose whether you want analyze your complete bug database or focus on the issues with the higher priority and/or severity levels.

There obviously are more metrics that can be gathered from your existing defect process and data, this is only my shortlist of personal favorites.
I invite you to add comments with your metrics and the way in which they’ve helped you improve your testing process.

There’s no replacement for good management. The leader of a team shouldn’t need artificial methods to know whether the group or one of its members is not performing correctly. Having said that, a responsible Test Team Manager should always track and analyze objective and empirical measurements of her team; these metrics will provide very little indications about the individual team members, but they will give invaluable information about the team’s performance.

What you measure is what you get…

Should you keep track for the amount of defects each team member reports? Do this only if you want to see large quantities of useless bugs in your tracking system.

How about counting the number of tests written by each tester? Measure this and you’ll get a lot of atomic test cases that could have been written as one, more complex and intelligent, testing scenario.

Can you measure the number of daily tests each engineer executes? By doing this you are telling your team that you don’t care about (important) bugs that are not directly related to their specific tests and may take a large amount of time to reproduce and understand.

The bottom line is: if you measure Quantity you are automatically compromising on Quality.

So what’s the alternative? How do you know if your team is doing a good job?
Let’s be sincere; if you are managing the team correctly and you understand what they are doing you already know.
If you were just brought into the Company, it will not be automatic, but after 3 to 4 weeks you’ll start getting the feeling, and by 2 to 3 months you better know for sure.

Good managers don’t need “a tool”, they use the MBWA (management by walking around) approach. They are interested in what’s being done, ask intelligent and relevant questions, and most importantly they are constantly learning (learning the products, the methodologies and the team).

So, should you stop measuring…? Absolutely NOT!

As a manager, you and your team have an objective that needs to be tracked and measured.

Most of us (Test Team Managers) are trusted with the objective of providing correct and timely visibility to the team in order to develop and release a product that answers our customers’ expectations; we are also expected to do this as close as possible to our targets for schedule, features and cost. [Author’s comment: since this is not an article about the objective of the testing team, even if you don’t completely agree with this definition try using it as an example for setting a set of metrics.]

There are 2 aspects to keep in mind about team performance metrics:
1. They should never be individual – a team is composed of members that compliment each other and thus should not be measured independently.
2. They should focus on the empirical outputs of your objectives – this is the hardest part of the metrics work; since objectives are usually abstract and metrics are extremely concrete. Let’s focus on this point.

How to set concrete measurements for abstract objectives?

Let’s take the objective definition we used above, and break it down into smaller pieces, then we will think of a set of metrics for each piece.

“…providing correct and timely visibility to the team…”

We can think of many different metrics for this part, these are two examples:
1. “Correct visibility” can be measured by tracking the percentage of rejected defects we get from development (vs. the total amount of defects reported). Doing this we are seeking information about the quality of our information.

2. To measure “timely visibility” we can add estimations of the time it took from the moment the bug was introduced into the product until the testing team detect the issue, calculating the project average. Take into account that design bugs were introduced long before they were coded into the product.

“…as close as possible to our targets for schedule, features and cost.”
To measure these 3 targets we can use the trivial metrics of planned vs. actual for timelines, features and cost measurements.

“…to develop and release a product that answers our customers’ expectations…”
Leaving the best for last )
How do we know if a product answers our customers expectations? I usually go for the simple solution of counting the amount of different issues reported from the field. Then I normalize this number, dividing it by the amount of issues previously fixed in the version, this way I get a metric called the escaping defect rate.

This metric provides a way to look deeper into the performance improvement opportunities of our team.
If we examine the issues reported from the field, they can be categorized into 2 groups:
a. Issues we missed in our testing
b. Issues we found in our testing but choose not to fix
The first group will point at places where we need to improve our detection methods (a.k.a. our tests).
The second group will provide information about places where we need to improve our ability to judge what is important to our users.

What to do with our metrics?

Once you have your metrics you can do at least a couple of things with them:

1. Create benchmarks for on-going improvement. Taking the measurements from the latest project and making sure that each following project will only provide better results.

2. A different approach is to look for Industry Averages for the metrics you are tracking and evaluate your team based on these numbers. Finding these averages is sometimes difficult (if not impossible!) but if they are known they can provide a great starting point.

The unstated objective of every Manager should be to constantly improve the effectiveness and efficiency of his team. Using well design performance metrics you can point to the places where the team has both the need and the opportunity to improve itself.

The biggest obstacle for working with Team Productivity Metrics comes from the natural human fear of making our weaknesses public. There’s no magic pill for this issue, only the knowledge that you are doing something right for the team and the Company. If your metrics program is good and provides results, you will see how in no-time other managers start leveraging your idea into their own teams.

Ever gone on a long car trip with kids? After some time they start growing restless and begin asking every 5 minutes (if your lucky) ARE WE THERE YET? ARE WE THERE YET? ARE WE THERE YET?!?!?!

I’ve had similar experiences in Development Projects. Close the release date we’d start getting calls from Product & Project Management asking us whether the product is ready to be released and if we can give them a date (even an approximation) for when will we be ready to go ahead. What usually happened in these situations was that Development proclaimed they were finished with all the features and now the QA was halting the project due to the bugs they were discovering in the product (like we were the ones who got them into the code in the first place…)

In my rookie days I used give the neutral answer that “we will be ready once we find and fix all the critical bugs”. Then after a couple of such projects I was able to find a way to give an answer that provided more information by giving visibility into the product via our test results and the statistics related to our bug detection and resolution convergence curves.

My experience is that bug detection behaves like a trend; if 2 weeks ago you detected 15 critical defects, a week ago 12 critical bugs, and this week 10 critical bugs, most probably you will detect between 6 and 8 critical bugs during next week’s testing. Bug detection trends can be represented by a curve that increases at the beginning of the project as you get more features and more people to start taking part of your testing operations, then it reaches a maximum peak when you get all your features and/or once you reach a critical mass of functionality to test, and then decreases as you start completing your testing scenarios and start running your regression tests.

If on top of this behaviour we also graph the fixing rate of the development we can then determine 2 things:
1. The time when development starts fixing more bugs than the testing team is finding per week. Even if this point by itself does not provide much vital information, it gives an indication that the project has reach the first turning point toward convergence.
2. By following this graph over a period of time we will be able to provide an intelligent forecast of the approximate time when we will be ready to release the product.

Even if the information on these graphs is not infallible, and even if we should always be prepared for the bug on the final CD before release, by using this data intelligently we are able to provide valuable information to the Organization regarding the time when the product will be ready to be released.

The main task of the QA team is not to be the Gate Keeper of the R&D, holding the product until it has been completely rid of bugs. Our Job is to give visibility into the process and product, and to provide information to all stakeholders that allows them to make their decisions concerning the product.

By making data our most valuable resource, we make information our most powerful tool!