PractiTest QA Blog

I was visiting friends and family last week in Costa Rica. Even though there is a growing software industry in there, my non-technical childhood friends only know that I work in “something related to computers” and for them that means I do everything from installing printers to writing the software for the space shuttle program. One of my biggest entertainment activities was to explain that there is a discipline called Quality Assurance, and that we work side by side with the development teams making sure the released products meet the desired levels of internal and external quality.

During a dinner conversation the father of a close friend asked me a simple question: How come there are bugs in software? From his point of view, if there are teams in charge of testing then there is no reason for the products to have defects. (Was he insinuating something about our job performance…?)

The answer to his question is economic in nature, and one of the biggest complaints of QA teams worldwide: “there are never enough time and/or resources to test the entire product”. Simply put, Organizations know that it is more cost-effective to release “some defects” than to test and fix all of them; especially if the company has “some control” on the number of bugs that will be discovered and the areas where they may appear.

This practice is not some sort of QA-Religious Sacrilege. It is the conscious practice of controlling and managing the number of Escaping Defects of the released application and using this measurement to grade and manage the quality of the product and of the development and testing processes.

Escaping Defects (ED) are all the bugs detected (and reported) by customers, regardless if they were also found as part of the testing process. The Escaping Defects Rate (EDR) is calculated by dividing them by the total number of bugs found in the product (both detected and escaping)

In the same way as different products require different levels of quality to be cost-effective each one should target its own acceptable EDR range. For example an Avionics System used on commercial airliners will aim to have no less than 0.001% and no more than 0.005% EDR after 36 months of been released, while an IM application aimed at high school kids will target between 1.5% and 3.0% EDR after 12 months in the market.

Some general rules of thumb regarding Escaping Defects:
- Enterprise Software will usually require lower EDR levels than Consumer Products
- New products and companies will be expected and allowed to have higher EDR levels than mature and established products and companies
- EDR’s behave like S-Curves over the course of time

Last but not least, since not all bugs are created equal EDR or any other defect statistic will not be able to tell the complete story about the quality of the released product and process; it is also necessary to monitor the nature of the defects been detected in the field. A defect on the main path of the product that renders an important part of the application inoperable will always have a bigger weight than a bug on a side feature. Just keep in mind that not everything resides in the numbers, justice is not always blind…

This means the job of the QA manager does not stop once the application is released. We must monitor the released product in order to evaluate and grade the quality of our product and process, providing feedback for a culture of on-going improvement.

Last week we reviewed the operational inputs of the software testing process. This week we should complete the picture by reviewing its outputs.

As part of our analysis we identified six channels that provide input to the testing process (Requirements, Scheduling, Designs, Risks, Defects, and Organizational Feedback); output channels are less in number but bigger in importance and complexity than their input cousins.

The main output channels for the testing process are:
- Testing Results
- Reported Defects
- Formal and Informal Feedback

1. Testing results are the first and most direct output channel.

When you report the scenarios that were tested and the results of these tests you are providing the equivalent of “military intelligence” to the development teams and project managers of your organization, allowing them to make timely adjustments and modifications to their tactical decisions and strategic plans.

The way you write the report is critical:
- Use an intelligent structure; start by giving summarized information with your analysis of the situation and not only the raw data.
- Stay away from endless tables with numbers and test statuses that don’t mean anything to anyone. Include summaries, percentages and convergence vectors with your numerical results.
- Use graphs, trends and extrapolations to help you give information to your stakeholders.
- When creating your reports, make sure each individual gains access to the information he requires: A project manager will want to see application coverage and risk analysis reports, but will not gain value from the status of specific tests. The opposite will be true for the developer in charge of the functionality covered by your test cases.
- Since report generation can take a lot of time, try automating as much of it as possible. Data gathering and graph generation can be quickly done using simple SQLs and tools like Excel Macros or report templates.

I will to leave the subject of test reports in order to move forward with the other outputs, but since this is one of the most important aspects of testing I will review it further on a future post.

2. Reported Defects are also an important output channel, and in some organizations the most widely used.

Don’t underestimate the importance of a good defect report.
- Select a good and comfortable defect management platform, one that will be useful to all stakeholders in your organization and will help you manage all the defects and data.
- Spend the time to create the appropriate template for your defects, one that will help your Organization in 3 major tasks:
(a) Defect categorization and prioritization – help the product team to decide if and when to fix each bug.
(b) Defect fixing – help the developer locate the error and correct it.
(c) Defect statistical analysis – help the QA and Project managers to understand if there are specific process or project issues that need to be corrected.
- An important thing that QA teams forget to include in their bug reports is an indication of the severity of the issue in the eyes of the end user. This information is not always trivial to the stakeholders who are not in direct contact with the application, running test scenarios in the way customers do.

3. Formal and Informal Feedback is the last of the common output channel.

These are all the comments, observations and other information aimed at correcting and/or improving the product or the process.

Examples of such feedback are:
- Usability reports providing user-perspective information about a specific product or feature to the product management or development departments.
- Defect summaries with an aggregated quality overview of a specific area in the application.
- Post-mortem process analysis reviews, enumerating the positive and negative aspects of the process in order to reaffirm or correct them respectively.
- Informal comments to developers and managers providing feedback on a feature or issues that may require further work.
- Cross group informal communication, mutually updating about the latest topics and issues affecting the different teams.

Summarizing:
Having seen many testing groups over the years and Continents I’ve reached the conclusion that most teams do their testing job more or less correctly, but nearly all of them fail in the job of communicating the information efficiently and effectively.

At the end of the day, doing the testing and failing to communicate the results is like not performing the tests at all, since the information will not reach the people who need it in order to make the correct decisions at the right time.

Back in school we learned that each process has inputs and outputs. Inputs provide the information and feedback needed to define the internal operations of the process, while outputs specify the deliverables that are handed out.

Some of the biggest process mistakes result from not identifying or failing to correctly map these inputs and outputs.

In the Testing Process the most common Inputs are:
- Application Requirements
- Project Schedule
- Software Designs
- Product and Project Risks
- Defects
- Organization Feedback

Outputs usually take the form of:
- Testing Results
- Reported Defects
- Formal and Informal Feedback provided to the rest of the Organization by the QA TeamLet’s focus now on the Testing Process Inputs:

1. Application Requirements are the most “trivial inputs”, and at times also the most difficult to find.
The main issues in gathering requirements are the lack of a formal and correct definition, the constant change in product scope (scope creep), and the lack of a single person or point of contact for definitions or clarifications.
Even with all these problems it is imperative to defined a set of application and testing requirements, and sometimes the QA team acts as the catalyst for the formalization of these definitions.
Remember that even if the nature of a project forces you to cope with changes in the requirements, at all times you need to make sure your testing coverage is based on the latest functionality that defines your product.

2. Project Scheduling is almost as important as requirements since one of the objectives of the QA is to provide timely information to all parts of the Organization. The project’s internal and external timelines (and their changes) define the time to start and stop specific tests in your plan.
Most importantly, you are expected to constantly analyze your test results in order to estimate if the application will be ready for delivery on time to the end user.3. Software Design helps you define how to test your application. Use the functional and non-functional design specifications to plan and structure the testing strategy starting from the complex and risky features and changes, which are the places where the most problematic issues have a bigger probability of appearing.

4. QA Risks come in 2 different flavors: Product and Project. Product risks are the ones I mentioned in the previous point and are related to complex and bug-prone features.
Project risks are the areas where a non-product issue may appear. Examples of these issues are lack of technology experience in a group, scheduling issues, geographical or cultural difficulties, or any other issueor factor that may harm or delay the project.
The QA manager needs to take these issues into account when planning the testing strategy and schedule. Important process risks should be reviewed and covered by tests in the same way as their product counterparts.5. Defects and defect-statistics are two of the main points of information for the QA Manager.
Understanding not only the bugs but their distribution and origins help you modify the tactics of the test plans in ways that provide better coverage in places where you need it.

6. Organizational Feedback is the last input channel and the less defined, it refers to all the “other information” that moves throughout the teams and helps you understand what is needed and where things are happening within the project, product and teams.
It is hard to write down how this information modifies the testing process, but any experienced testing manager understands the strength and effects it has on the complete development process.Process inputs serve as a guide to the QA Manager, who constantly needs to asses the situation and make modifications to the plans in order to assure the best possible coverage and timely information. Ignoring or failing to react to this information would be like driving at night with the lights off and your eyes shut, hopping the plans you made before you left home will safely deliver you to your destination.

Some testers and QA managers won’t admit they use informal testing as part of their QA routine. In some circles this testing approach is considered a waste of time or an unprofessional way to perform a testing task.

I think this is a misconception, and a knowledgeable tester should enjoy from the advantages of both QA methodologies.

Whenever I start to plan a new testing project I make sure to include both formal and informal testing activities into my testing schedule. Here is a simplified and generic example of how I do it:

Stage 1: Requirements Definition
Testing Objective: Assure that all the application requirements are properly defined by the customer and/or project responsible party.
Testing Approaches:
(1) Informal walk-troughs of the requirements where the tester takes the role of “Devil’s Advocate” to make sure that all corners of the application are covered by the current definitions.
(2) Formal review of the requirements to assure that the structure, language and information provided is correct and consistent.

Stage 2: Software Design
Testing Objective: Assure that the design covers all the necessary aspects of the application such as functional consistency, security, backwards compatibility, scalability, testability, etc.
And additional objective for the tester during this phase is to gain a better and more concrete understanding of the application in order to write his formal test plans.
Testing Approach:
(1) Informal design review meetings with the solution architects and application development teams to go over the high and low level design documents of each part of the system.

Stage 3: First Delivery to QA
Testing Objective: Provide initial feedback to the development team regarding core aspects of the application such as stability, usability, consistency and other elements that may require major coding re-work.
Calibrate the QA Testing Plan according to the initial status of the application.
Testing Approach:
(1) Formal acceptance test cycle, covering all the major functional and infrastructure areas of the application. Assure that no major bugs or blockers will prevent the continuation of the tests before engaging the whole team on the task.
(2) Informal exploratory testing of the application, to get fully acquainted with all parts of the application and quickly detect most of the low hanging bugs in the system.
(3) Formal initial testing cycle, performing the first complete swipe of the application or at least of all the new functionality for the specific project.

Stage 4: Periodical Deliveries and/or Builds to QA
Testing Objective: Provide input and feedback to the development team regarding the status and progress of the application, while reporting all defects found in the system.
Monitor the progress of the project, providing feedback regarding its major assumptions and risk (e.g. timelines, technological stability, regression status, etc).
Testing Approach:
(1) Formal build acceptance test, making sure the delivered product is stable enough to enter the testing cycle and engage the whole team on the task.
(2) Formal testing cycle, covering all the areas of the application based on the test plans written with the knowledge gathered from the requirements and design documentation and reviews.
(3) Informal testing activities such as bug hunts and/or role playing testing in order to catch bugs in places and scenarios that were not covered by the formal cycles. These activities should be timed based on the stability of the application and the progress of the project in order to assure that they are efficient and effective.

Stage 5: Final Acceptance Tests
Testing Objective: Assure that the product answers all the customer’s or end user’s requirements based on own their testing scenarios.
Testing Approach:
(1) Formal testing based on the customer-defined scenarios.
(2) Informal testing based on customer scenarios and role playing testing if possible in the customer’s environment and with their duplicated data.

Stage 6: Postmortem Analysis
Testing Objective: Review the recently completed development and QA processes, gathering all positive and negative feedback from the participants and customers in order to leverage the knowledge and experience learned on this project on to the following ones.
Testing Approach:
(1) Informal questionnaires and cross-group debates, analyzing all the separate stages of the process in order to create a list of lessons learned and points to re-enforce for future projects and processes.
(2) Formal analysis of statistics gathered throughout the whole process.

My experience is that informal testing techniques are not only valid but necessary during some stages of the development and testing lifecycle, and they provide the most effective way (in planned conjunction with their formal counterparts) to correctly cover an application. In most projects this balanced mix of formal and informal methodologies will provide the best coverage for your system under test.