PractiTest QA Blog

Almost 20 years ago Boris Beizer stated what became known as the Pesticide Paradox:
“Every method you use to prevent or find bugs leaves a residue of subtler bugs against which those methods are ineffectual.”

In plain English this means that as you run your tests multiple times, they stop been effective in catching bugs. Moreover, part of the new defects introduced into the system will not be caught by your existing tests and will be released onto the field.

This principle (or paradox) came up during my conversations a couple of times lately.
Once when evaluating a company’s approach to automation, where they created a large suit of tests and assumed it would continue catching all new bugs for eternity. Another time, working with a different team who was looking for the reasons their existing manual testing suit was not detecting all the bugs before releasing their product to the field.

The truth is that test suits require constant maintenance and updating, regardless if they are automated or manual.

There are a number of reasons a perfectly good suit of tests will stop been effective over time:

1. The Practical Impossibility of Testing all Possible Scenarios.
Even simple applications require an exaggerated and impractical number of tests in order to verify all possible scenarios and data combinations. This is why we use the help of methodological tools such as equivalent partitioning and model-based testing, but still this is not enough.
In the end of the day most teams will use a risk-based testing approach to create a sub-set of scenarios and data-sets, and then use the escaping defects found in the field after the initial release in order to calibrate and patch any holes that may have been left the suit.
This means that at the end of the day we don’t test everything.

2. The functionality of the application changes over time.
If we introduce new features to the product it may seem trivial that we need to write tests for them. It is less trivial to remember that we also need to modify the tests for the existing features, even if they are only slightly modified by the new additions.

3. We (humans) tend to be especially careful only on places where we feel imminent danger.
What does this mean?
Simply that developers will be extra-careful in places where testers found bugs beforehand, but on the other hand they might not be so careful in places they “feel comfortable” with.

So what do we do with this? How do we assure we are working with an effective and efficient suit of tests?

The key rule is to be objective and to constantly keep reviewing the state of things. In practice I recommend the following:

1. Keep track of product changes and their indirect effects in your application.
The direct changes are trivial, but make an effort and do all the structural and functional connections, then think of the new scenarios you need to write to cover these changes.

2. Discontinue tests that are not effective.
Too many useless tests may be an overhead more than they can help your process.
For example, if you have 10 tests that cover the same area and none of them have detected bugs in an important number of cycles (the number is up to you!), then review them and cut their number down.
My rule of thumb is that if test has not reported a bug in the last 5 runs, I add it to my review list and I start verifying its importance and weighting whether I should keep it or move it to my test archive.

3. Modify your test data.
This one is trivial but we tend to forget it.
Many bugs in our products are data-specific, this means that (specially in our automated suits, but not only in them!) we should continually increase and/or modify our test data. This may mean getting more samples of demo databases or creating new batches of social security numbers or whatever that means to your system, but make sure to add randomness to your tests.

4. Last but not least, don’t put all your trust and weight on formal approaches only.
Use at least one type of informal testing per cycle (or at least per release).
Either execute Bug-Hunts, or Exploratory Testing, or whatever other approach you want to use. Adding the human approach to any testing cycle will automatically increase its effectiveness.

To summarize:
It is both naive and dangerous to assume you can create the “ultimate testing suit”, one that will discover all bugs once and for all.

Even if you create a suit that has a very high coverage percentage and bug discovery success rate (calculated based on the level of escaping defects), you should not lay down your guard.
You need to continue reviewing and verifying your test suits and scenarios regularly, and adding testing factors that will assure you continue testing your system in an effective and efficient way.

Development Organizations are in constant change.

A change in the Product Marketing strategy may lead to splitting the development baseline into individual and separate products, or the growth in sales of a product may even support the decision to split an Organization in to separate Business Units.
Opposing change is like trying to stand in the way of Evolution, it may take longer to achieve it but at the end of the day Change will take place (and hope you don’t share the fate the Dinosaurs).

A question that arises as part of Organizational and Structural changes within an R&D (Development and Testing) group is when should we split our Issue and Test Management System into independent environments?

There is no text-book answer. There are Pros & Cons to having one large-concentrated project and/or many individual-smaller ones; and the decision should be made for each case separately.

Having one large and concentrated project has multiple advantages:

1. A single place of work for everyone assures you are always looking at the correct set data.

2. When working with one environment you can migrate and share data between projects, for example a bug that is detected in one project can be assigned to a separate team for fixing and handling.

3. When all your data sits in one place you have the ability to aggregate and compare the information from all projects in order to get comprehensive reports (specially valuable to Higher Management)

4. Lastly, one centralized project makes it easier to implement and enforce a unified working process and methodology.

On the other hand, having a project with lots of unrelated data has its disadvantages:

1. When you have tons of data in your environment it is harder to easily find the information you require to work.

2. If you need to force a system to serve the needs of many teams it will automatically be optimal to none of them, as it needs to accommodate many more constraints than if it was to serve a single or a smaller number of teams.

3. My personal favorite (and you may disagree with my psychosis), having individual projects allows you to start fresh as many times as you want. I always enjoy the opportunity to “clean the desk” and leave behind all the mistakes of the past, looking at the future with a new and fresh perspective.

In order to take a practical approach, you need to review what are the individual needs of your organization and make a decision for yourself (with your respective stakeholders, obviously!). Over the years I realized that 2 main criteria usually help me decide what path to take: The level of Data Sharing, and the Incompatibility of the Processes.

1. If separate projects have important amounts of shared information (i.e. Issues that flow between Teams, Tests that are reused between Projects, etc) you should try to keep them under a single Management System.

2. If the working processes of the teams (dictated by external constraints, the nature of the project or even cultural differences) are too incompatible you should try to separate the systems into individual environments that will serve the needs of each team.

These are only rules of thumb and they need to be reviewed together with the rest of your Organization’s individual (and sometimes singular) constraints, but 8 times out of 10 they will be your main guiding points.

I could not finish this article without referring to some of the most common WRONG REASONS why some Organizations decide to split their working environment into individual projects:

1. Geographical dispersion of the teams.
This used to be a limitation in the past, and is still a real limitation for some Platforms in the market today (that continue using the wrong technological approach). But today there are enough platforms designed and developed in order to work seamlessly for distributed teams regardless where they sit around the world (if you are looking for one of these systems you can check QuackTest out).

2. Amount of data in the system.
If you have too much data in your system you should think about
(a) making some order and archiving (or erasing) the stuff you don’t need,
and/or
(b) working with a system that can help you accommodate the amount of information you need to manage.

To summarize, when an Organization changes you need to make sure your infrastructure still suits its needs.
Having said that you should not jump into breaking down your project without understanding if this is the right move for you. Take into account that it is very easy to break a system down into 2 or more instances, but it is extremely difficult (some would say close to impossible) to merge 2 or more systems into one.

This week the subject came up again at a customer: “why do we need a separate bug field for Priority and Severity…”?

To make a long story short, the Dev Lead kept saying that he only looks at the Priority field and so we shouldn’t waste time writing down the Severity; then (to emphasize his point) he said that 90% of the time Priority and Severity have the same value and where they don’t it was only since the QA wanted to exaggerate the severity of some Bugs…

Without getting into the outcome of the discussion, I think that in most projects Priority and Severity should be 2 separate fields.

1. Severity refers to the Bug and how it affects the User’s interaction with the Application. Priority refers to the Project and how urgent it is to solve the Bug in the context of the work been done.

2. Severity is objectively set based on the direct and indirect repercussions of the bug, together with the probability of its occurrence. Priority is set based on changing project factors such as where is development and QA focused at the present time, how many more defects do we think we will fix before release, how important is this bug for a specific customer or objective, etc.

3. Severity is usually a static field, the only reason to modify it would be if we learn something new about the bug (such as possible workarounds, additional user scenarios, etc), or if the functionality of the application changes in ways that affects the bug.
Priority should be dynamic, to be revised and updated as the project progresses and we want to shift the focus of our efforts into specific product areas, and as we start reaching the end of the schedule and we need to make sacrifices about what will be fixed for the current version and what will be pushed to future ones.

The Priority of a bug should be set and modified in agreement by the project stakeholders based on the strategic and tactical decisions of the project. In most cases, this is done during a regular (daily, weekly, etc) meeting where they review and evaluate the open bugs in the system.

An additional gain of correctly working with both fields is that as we start a new project and we need to review all the bugs left opened from the previous one we have a way to categorize and organize our list based on its objective severity.

Having said all the above in some cases, and specially on small or simple projects, one field will do for both values. This happens mainly when all bugs are fixed based on their Severity and thus Priority becomes redundant.

Regardless of how you organize your Defects and/or your Project. You need to make sure to capture as much information for the bug as possible. This will help you in your present project, and it will also provide you with valuable information for future ones.

In many R&D organizations the QA takes the role of the Customer Advocate, testing the system and even arguing in favor of fixing defects based on their internal perception of how customers use the product.
In some companies Test Engineers even take the role of Users during Internal Design Reviews.

Why does this happen?
Your guess is as good as mine but I think this is an issue of perception.

If testers already have an end-to-end perspective of the system, and they take the role of users when testing the product, this makes them good customer advocates, right?
WRONG!
If most testers have never even talked to a user, how can they advocate for them???

I think that QA is a good candidate for internal Customer Advocate, as long as we make sure they have the tools and experience to fulfill this role.
There are at least 3 ways to provide our Engineers with the knowledge and experience required to represent Users.

1. Place them closer to users within the Organization.
Do this by letting them take part of Customer Support activities and by supporting sales teams during pre-sale cycles, it can be as simple as instructing the Engineers to talk and learn from these customer-oriented organizations about the main needs, pains and comments coming from field.

2. Take Engineers out into customer facing activities.
For example, by sending them on-site to work with customers on complex installations or advanced implementations of the system. Another good opportunity is during Beta Programs, Customer Trials, or any other activity where the R&D interacts with users face to face (preferably on-site).

3. User conferences
Not all companies have them, but if you do this is the easiest way to gain access to a big number of users.
During conferences it is important to have an effective way to interact with the users, you can do this by arranging meetings up-front (with the help of the sales or support teams) or even by setting up a Demo Area where the QA can demonstrate the new or up-coming functionality.

At the end of the day the QA needs to take the role of Customer Advocate, there’s no way around it. If so, it is important to let your engineers do this right, by giving them the tools and knowledge required for this task.