PractiTest QA Blog

I make a point of not making important decisions in the heat of the moment. For example, when I disagree with someone on a professional level and tones start getting hot during a meeting, I ask for a timeout or a change of subject and to re-take the issue after a 24 hours period of perspective. In 90% of the cases this time will help me and my colleague to better understand the other’s point of view and reach an understanding that allows both of us to walk out of our argument happy.

Still, there are times during a development project when you need to make a decision on-the-spot with no way to “snooze” your assessment. One of these situations happens around the end of the project, during what many of us call Release Decision Meetings; where representatives from all departments get together to provide their inputs and jointly decide whether the product is ready to be released to the public.

I’ve been part in many of these meetings as a QA Manager. These are the times when on the one hand you are pressured to release the product NOW (or as soon as possible), but on the other hand you carry a big part of the responsibility for the stability and the quality of the final product been released.

How do you make sure to be objective and not to make mistakes at these times of pressure? One ways is by having a Release Criteria list that was defined and agreed up front by all the project stakeholders.

What is a Release Criteria? Simply put, it is the list of Objective Prerequisites that should be met by the product and process in order for us to feel confident about releasing it to the public. The most important aspect of the Release Criteria is that it needs to be reviewed and agreed by all product stakeholders prior to the Release Meetings and preferably during the early stages of the Development Processes.

The release criteria will never replace the Release Decision Forum, as this group of people needs to take into account all objective and subjective aspects surrounding this decision; but it will help you (the QA Manager) to provide an Objective Evaluation based on the parameters that were considered important before we started being influenced by external and maybe inappropriate factors.

What comprises a good Release Criteria list will vary from product to product and from company to company, but some basic items that should be included in it are:
1. Percentage of the original required functionality that needs to be in final the product.
2. Testing coverage level – percentage of the tests that need to be run on the application from the original testing goal.
3. Pass rate for the testing coverage –percentage of the executed tests that need to be successfully run without falling into blocks or failures.
4. Number of critical defects still open on the product.
5. Maximum defect detection rate for the last month, week, or x amount of days– based on the assumption that enough testing is still being done on the product.
6. Minimal load tolerance (if applicable).
7. Minimal amount of time the application has been running continuously on the testing and/or staging environments (if applicable).
8. Minimal amount of time the application has been on a Beta Testing trail (if applicable).

It is always recommended to start updating and publishing the status of the project with regards to the Release Criteria as early as logically possible. If you have Kitchen Monitors in place you should consider adding a page with it the to your running template.

As with all information generated by the QA, using the Release Criteria wisely will increase the amount of value you provide into the process.

Ever gone on a long car trip with kids? After some time they start growing restless and begin asking every 5 minutes (if your lucky) ARE WE THERE YET? ARE WE THERE YET? ARE WE THERE YET?!?!?!

I’ve had similar experiences in Development Projects. Close the release date we’d start getting calls from Product & Project Management asking us whether the product is ready to be released and if we can give them a date (even an approximation) for when will we be ready to go ahead. What usually happened in these situations was that Development proclaimed they were finished with all the features and now the QA was halting the project due to the bugs they were discovering in the product (like we were the ones who got them into the code in the first place…)

In my rookie days I used give the neutral answer that “we will be ready once we find and fix all the critical bugs”. Then after a couple of such projects I was able to find a way to give an answer that provided more information by giving visibility into the product via our test results and the statistics related to our bug detection and resolution convergence curves.

My experience is that bug detection behaves like a trend; if 2 weeks ago you detected 15 critical defects, a week ago 12 critical bugs, and this week 10 critical bugs, most probably you will detect between 6 and 8 critical bugs during next week’s testing. Bug detection trends can be represented by a curve that increases at the beginning of the project as you get more features and more people to start taking part of your testing operations, then it reaches a maximum peak when you get all your features and/or once you reach a critical mass of functionality to test, and then decreases as you start completing your testing scenarios and start running your regression tests.

If on top of this behaviour we also graph the fixing rate of the development we can then determine 2 things:
1. The time when development starts fixing more bugs than the testing team is finding per week. Even if this point by itself does not provide much vital information, it gives an indication that the project has reach the first turning point toward convergence.
2. By following this graph over a period of time we will be able to provide an intelligent forecast of the approximate time when we will be ready to release the product.

Even if the information on these graphs is not infallible, and even if we should always be prepared for the bug on the final CD before release, by using this data intelligently we are able to provide valuable information to the Organization regarding the time when the product will be ready to be released.

The main task of the QA team is not to be the Gate Keeper of the R&D, holding the product until it has been completely rid of bugs. Our Job is to give visibility into the process and product, and to provide information to all stakeholders that allows them to make their decisions concerning the product.

By making data our most valuable resource, we make information our most powerful tool!

We’ve all heard of Police SWAT teams. These are Special Forces used when a situation takes place requiring interventions outside the capabilities of the normal officers and patrollers. Still, most people don’t know the meaning of the name SWAT, Special Weapons And Tactics.

There are times when a Special Weapons & Tactics team is needed also within the R&D Organization; when we seek a rapid and effective solution for a “special project” and there is no regular team that can deliver it by itself using their normal and/or known procedures. This is when an organized group of Testers and Developers from different teams can be of help, each contributing the tools & tactics from their day-to-day practices.

R&D SWAT teams are different than their police counterparts, they are usually created only after the need is identified, and their members are selected based on the specific characteristics of the challenge at hand.

The use of these teams comes at a price and thus you need to be careful when using them.
Following are some tips to keep in mind:

1. Don’t abuse them. If you start using SWAT teams all the time they loose the effectiveness that comes from the fact that team members feel they are working on a task that is out of the ordinary, both in nature and importance.

2. Be very strict appointing the team, and then let it loose. You need to trust the team and especially its leader to make wise decisions; once you have done this STEP BACK and keep out of the way to let them work and solve your problem.
If you feel that you cannot do this then try been part of the team; just understand that you will need to obey the rules and work EXCLUSIVELY on the problem for the duration of the task.

3. Create a synergistic team (where the integration of the elements is greater than the sum of the parts). This means getting people who (a) compliment each other, and (b) can work together and communicate. It is important to have people from as many expertise fields as possible, as long as they have something to the task.

4. Define a deliverable but don’t limit the methods. Each team member should know what they are expected to deliver, but they should also know that the how is up to them.

5. Limit the timelife of the task.

6. Support the team. The rest of the R&D Organization should know that whatever your SWAT team is doing takes precedence over any other task. This means they should help and cooperate with whatever these guys ask and need.

7. Publish the results. SWAT teams should not work in secrecy; both success and failures need to be shared and analyzed by the whole Organization in order to learn from them. If the team does not succeed make sure this is not taken as a personal failure for any of the members.

8. Reward accomplishments. If the out-of-the-ordinary job is accomplished make sure to reward accordingly. People should fight for a chance to prove themselves in these teams.

My personal experience with SWAT teams has been very positive over the years, especially around situations that had very strict time limits around complex and risky deliverables (e.g. solving hard issues happening on production Customer Sites). I recommend using them wisely since their abuse will only make your team less motivated and your difficult tasks unachievable.