PractiTest QA Blog

Bug Hunting is becoming a common practice form many testing organizations world wide; yet some test managers wrongly feel they go Hunting when their testers informally play with the application in order to find “border-case defects”.

Bug Hunts are Informal Testing exercises; this should not be mistaken with playing with the system without a purpose or objective. In order to achieve something (and not waste your time!!) during these activities you need to follow a specific methodology, perform planning and preparation actions, and monitor and control the process throughout its execution.

Even if there is no written book (at least that I am aware) on Bug Hunts, I follow a process that I caught some years ago during one of the STAR conferences.

I plan Hunts based on Pair-Testing and Soap Opera Scenarios, combined as a tournament between teams.

Pair Testing is when you take 2 engineers and they work as a team on 1 computer (or environment). The idea comes from pair programming with all its known advantages such as knowledge sharing, brain storming, and positive pressure from working with a partner. I’ve had very good results pairing engineers from different teams; especially when taking development engineers and paring them with test engineers, leveraging the advantages and points of view from both sides of the process.

Soap Opera Scenarios are relatively short end-to-end scenarios that take the system from beginning to end of a complex operation on a fast (and sometimes exaggerated) chain of events. Working under extreme conditions we are able to find issues that we missed during our controlled scripts and scenarios.

The Tournament activity is where the human factor comes into play. You want people to be motivated, and what better motivation for an engineer than a price… I keep track of many statistics like the number of original bugs detected per day, the most serious bug detected, the worst system crash, etc; and give a price to the pair who exceed on each category (we usually give away t-shirts and on one extreme case we even gave an iPod!), in any case it is not the price but the spirit that makes the difference.

There are some rules of thumb I give QA organizations before they start planning their Bug Hunts:

1. Make sure you’ve reached a minimal application maturity level; if it is still too easy to detect critical bugs or the system keeps crashing all the time you may be ahead of your time.

2. Work on testing environments with real life data. Your tests need to be far from sterile in order to simulate a real working environment.

3. Create a balanced activity schedule. I work based on 2-day bug hunts, where each day starts with 4 hours of Exploratory Testing around a specific application area (each team or two working on a different part of the application); and during the second half of the day we run our Soap Opera scenarios, where each team receives a user profile and a list of high level tasks.

4. Manage the activity using a tracking system. Any bug tracking system will help you keep track of the issues detected by the teams; if possible use a dashboard to make sure all team members are updated on the position of the rest of their peers.

5. Have a bug referee, she will decide whether the findings are real bugs and also stop duplications from been reported and counted.

6. Create anticipation by having internal teasers and “publicity campaigns” ahead of the activity.

7. During the hunt generate an informal atmosphere by playing music, giving each team bells to signal whenever they find and report a new bug, bringing pizzas and sodas, and using other out-of-the-ordinary things that will create the feeling of a special occasion.

Bug Hunts are as much about the right atmosphere and state-of-mind as they are about the methodology and scenarios you run. Make sure you and your team have fun and it will surely generate the outputs your expect it to produce or more.

Imagine waking one day with a terrible feeling in your left leg. You go to your Doctor and his first question is “where does it hurt?” He examines you, asks all sorts of related and unrelated questions, and prescribes pills for tendon inflammation.

A couple of days later you are feeling a lot better and you meet a friend for lunch. After talking with your friend in the restaurant for about 10 minutes a waiter comes to the table with two glasses of water and asks for your order. You were busy all this time and didn’t even look at the menu, so you ask the waiter what should you eat? He starts by asking you what type of food you like and what are you in the mood for, he then offers you a couple of interesting options and you decide to go for the beef. After following his advice you are amazed by the quality of the food and the service, so you decide to leave him a large tip.

What do a Doctor, a Waiter and a Tester have in common? All of us provide services for our customers!

In Testing we sometimes forget we have customers, and I am not talking about the end users of the product. Our customers are the people within the Organization who require our testing services; quoting James Bach: “The ultimate reason testers exist is to provide information that others on the project need to create things of value…”
Our customers are the Development Team, Product Managers, Project Managers, Customer Support, Higher Management, and all the rest of the people who rely on our services to provide them with Visibility and Information around the Quality of the Product and the Process followed to develop it.

What do I mean when I say we forget about our customers? Many times we plan and run our testing projects without asking our internal customers where does it hurt?
When was the last time you asked the following questions before starting a project?
- How is the development project structured and when do you require feedback for each individual feature so that you can work on it most effectively?
- What are the main risks areas in the application that we want to clear first?
- What are the most important features in this release, those we want to test most thoroughly? What product areas will not be very important for our users, so we can lower their testing priority?
- What support information do you require with the product to help our end users deploy faster and easier?

We don’t expect our customers to have many inputs into how to test the product in the same way a professional Chef does not get cooking strategy lessons from his clients. Still, like in the case of a Doctor, the only way to do our job effectively and to satisfy the needs of the Organization is by understanding what’s needed and important, consulting with the different stakeholders involved and asking them how we can help them to do their jobs better.

We provide a service and the only way to provide the correct one is by asking our customers where does it hurt?

I’ve been working lately in a couple of projects where the Make vs. Buy dilemma around testing tools has been popping up from multiple fronts.

In order to correctly solve this predicament we need to understand the reasons and the situation where it usually rises within most development and testing Organizations.

From my experience, the flow of events is as follows:

1. In the beginning the Organization uses Word and Excel to manage their process. This includes separate but linked documents or spreadsheets to manage the tests, bugs, risks, etc.
At one time, usually after a medium or large product crisis, someone reaches the conclusion that the Testing Team has outgrown these tools and it now requires a more professional platform to manage their assets and processes.

2. A basic search of the commercially available tools and platforms reveals that the alternatives are to either spend a couple of hundred of thousands of dollars on an Enterprise Platform that will need to be deployed, customized and maintained to match the Organization’s needs; or to buy a number of cheaper products that will require the team to compromise on the functionality or to internally develop their own integrations and customizations (on top of these tools) to supply the features they need.

3. It is usually at this point that the alternative to develop an internal platform comes forward. The rationalization is the following:

If the Organization will need to invest a large sum of money to buy and customize the testing platform;
AND
If regardless of the application or platform chosen, there is no single product that will meet the needs 100%;
AND
If the Organization has the internal knowledge to develop good software products in-house
THEN:
Why not develop the testing platform in-house, creating a product that meets all the Organization’s needs at a price that is close or lower to what the Commercial Product would charge?

The assumptions above sound pretty logical, and that is why we see many inexperienced and overexcited teams jumping into these projects…

Here is where experience comes into play. There is no magic answer to whether a company should make or buy their testing platform; although after many years in this business I have seen a lot of companies that developed their own platform (and invested more than a couple of development man-years in the project) and then decided to acquire a commercial platform and to migrate their data and process.

The main reasons they tend to do this are the following:
- The total cost to develop and maintain the platform ends up being close or higher than the cost of buying and deploying a commercial platform.
- The commercial platforms have more readily available features and integrations than most in-house projects are able to develop.
- The externally developed tools keep improving their features and actualizing their technology, methodology and processes; while the in-house tools only maintain the functionality based on what the team is currently doing.

Overall and contrary to the initial assumptions and estimations, creating the tool in-house will most times be more expensive and provide less functionality than acquiring an external tool.

I don’t claim that there are no successfully developed in-house testing platforms, I only point at the fact that the hidden costs of creating and maintaining the system may be bigger and make the process less cost-attractive than buying a good system up-front.

Take into account that there are cases where there is no alternative and the Organization needs to opt for an in-house developed platform.
Classic cases are the following:
- When the system under test is extremely complex and technologically advanced requiring a specialized set of tools that is not available commercially (as is the case with many embedded and real-time systems).
- When regulations or other external factors do not allow the Organization to compromise on the functionality required from the testing platform, as is the case with extremely regulated industries such as the aeronautics and pharmaceutical industries.
- When the system uses legacy technology that is no longer supported by any vendor.

To conclude the subject; there are no magic answers but on this issue most of the time you will want to go for a commercial tool and compromise on the price and functionality you will receive. There is no optimal product (at least not for now), but the alternative of developing and maintaining one in-house is more complicated and less feasible than what many of us imagine based on our raw assumptions and lack of experience in the matter.