On-going improvements are usually sitting quietly under our noses…Taking a closer look at our bugs and the information available in them we can come up with very strong upgrades to our testing process.
Following are 4 metrics you can collect easily during your regular bug lifecycle that can provide good leads into places where you can greatly improve the quality of your testing process:
I. Defect Density (by product area, testing phase, development team, etc)
Goal: Identify risk areas where you need to do more testing.
How to track it? Use one or more list fields to define the product area or module for each detected bug.
Possible Improvement(s): Analyzing a concrete statistical view of the risk areas in the product you can decide whether to modify your testing assignments, schedules or plans to better cope with these risks.
(1) Make sure you are not pointing a flash-light at a bonfire calling for the fire department, places with a lot of development work are likely to generate a high number of bugs.
(2) This metric can be used within the process to modify your current project, or during the project post-mortem to provide feedback for future projects.
II. Defect Rejected Reasons
Goal: Understand why your testers are wasting time reporting the wrong bugs, rewriting their bugs better, or arguing with developers unnecessarily.
How to track it? Create a list field with the possible reasons for rejecting bugs, make this field mandatory whenever an issue is set to status rejected.
Rejected reasons vary between organizations, but the list will always include values such as: “Missing Information”, “Duplicate Bug”, “Not a Bug” and “Unable to Reproduce”.
Possible improvements: If the team has a high rate of rejections look for the main problematic reasons and implement corrective actions. These actions may range from creating a new bug description template, to training engineers on how to search for duplicates bugs in your existing repository, or any other action to treat the source of the problem.
(1) Sometimes you will need to look for specific team members.
(2) Make sure you have a problem before implementing a cure; it is natural to have rejections, just make sure they are keep at acceptable levels.
III. Escaping Defect Reason
Goal: Map problematic areas where defects are been constantly reported by customers.
How to track it? Whenever a bug is reported from the field register whether it was previously detected by the testing team or not . If it was detected why was it left unfixed, and if it wasn’t found explain why.
Create a list field to register this information. Possible values for this list are: “Found – Low Priority”, “Found – High Risk”, “Missing Test Scenario”, “Border-case Scenario”, “Missing Environment”, “Border-case Environment”, etc.
(a) Increase your testing scenarios and/or environments with information from the field.
(b) Study application usage patterns in order to become a better “customer advocate” and provide better priority and severity information to your tests and bugs.
Notes: It is unrealistic to detect and fix all the defects, but we need to make sure we are not missing some of those we would really like to find in our process.
IV. Corrective Test Actions
Goal: Correct cases where bugs were caught by chance or to late in the process in order to be fixed without incurring in delays or risks.
How to track it? Create a Memo Field called “Corrective Test Actions” where testers or developers fill information that should be considered in order to create new test cases or modify existing ones.
Whenever a bug is detected by chance or too late in the process the responsible tester or developer should provide suggestions or comments indicating what needs to be corrected in the process or test scenarios.
Possible Improvements: At the end of the cycle or project all the bugs that have data on the Corrective Test Actions field should be examined and their comments be submitted as part of the feedback to the Test Manager and to the general post-mortem process
Note: There can be some overlap between the this metric and the previous one (Escaping Defect Reason).
A colleague once observed that “not all bugs are created equal…” and thus you can choose whether you want analyze your complete bug database or focus on the issues with the higher priority and/or severity levels.
There obviously are more metrics that can be gathered from your existing defect process and data, this is only my shortlist of personal favorites.
I invite you to add comments with your metrics and the way in which they’ve helped you improve your testing process.
Practitest is an end-to-end test management tool, that gives you control of the entire testing process - from manual testing to automated testing and CI.
Designed for testers by testers, PractiTest can be customized to your team's ever-changing needs.
With fast professional and methodological support, you can make the most of your time and release products quickly and successfully to meet your user’s needs.