PractiTest QA Blog

This topic may seem simple, but I’ve been asked about it a couple of times in the last month, and I also saw it in multiple questions on QA Forums, SoftwareTestingClub, and other places where I go visiting once in a while.

Let’s start by admitting we have a problem in the Testing Community. Not only do we call the same process by multiple names, but sometimes some of also use the same name to call different processes. This is starting to sound like The Bible’s story of the Tower of Babylon, but it is a fact and maybe part of the reason for the confusion.

So instead of trying to propose a Definitive Naming Convention I will present mine and I hope that it will make sense (pay attention to the content and if necessary disregard the names!).

Since this can become a very long post, I will condense it by naming each Test Type and then answering 3 simple questions:
- When (do we perform it)?
- What (is the content of the test)?
- Why (is it important)?

1. Development Integration Testing
WHEN? Before each build is delivered to QA.
WHAT? End to end scenarios covering the main path of the application. Should not last more than 30 min to 1 hour.
WHY? Assure that development does not “throw their builds over the wall” without making sure they meet the minimal stability requirements.

2. Build Acceptance Test
WHEN? The minute QA receives a build from development and before all the team starts deploying it on their testing environments.
WHAT? A 1 to 2 hour intensive test suit that comes to assure the build is good enough for the whole team to commit to it. This is a great candidate for automation (and if so, it can be run each day as part of nightly build system).
WHY? Even after developers perform their Integration Tests it is not always enough for the QA to start testing it. We need to make sure there are no blockers or other stability issues that would prevent you from executing your test plan.

3. New Feature Testing
WHEN? On each build that includes new features or functionality.
WHAT? Test the new functionality as deep and wide as you can, go over all the scenarios and the functionality you can think off.
WHY? The trivial answer: new features come with a high risk of containing bugs and thus need to be thoroughly tested.

4. Bug Verification Testing
WHEN? I do it on each new build, together or immediately after the New Feature Testing. There are some organizations where they do it at the end of the project but I think this is risky.
WHAT? For each bug that was fixed (or in some cases a family of bugs) you need to test the reported reproduction scenario, and then with the information provided by the developer regarding the root cause of the defect and the fix he implemented you should test additional scenarios that may still have bugs or new bugs that may result as a side-effect of the fix itself.
WHY? Similar to the New Feature Testing, places where people touch the code (even if it is done to fix a bug) can cause additional defects.

5. Regression Testing
WHEN? Once the new features of the product reach a minimal level of stability and you want to start testing and looking for bugs on other less-trivial areas.
WHAT? This test is the most complex suit to design. On the one hand it cannot be too extensive since you never have enough time to run all your tests, on the other hand it should cover all or most of the application and provide a good indication that the AUT or the specific Area that is being tested is CLEAN and stable.
WHY? Software Development is very risky, changes in one place can have unexpected negatives effects in other areas, and it is our job to lower the risk of these bugs being released to the field.

6. Sanity Testing
WHEN? When you need to check the application on a high level and don’t want to or cannot run a Regression Test (i.e. after all tests are done and you want to check the final CD that was burned and sent to mass-production).
WHAT? Similar to the Regression Testing in the fact that you choose specific scenarios that cover all the AUT or application area but shorter and based on the highest risk factors.
WHY? Same as Regression Testing, but here you are required to make a risk judgment call and compromise part of your testing.

7. User Acceptance Test
WHEN? When the user receives the final product in his environment, but sometimes since the testing team already has these scenarios up-front they can run before they release of the product.
WHAT? What the user decides to test. Usually End-to-End scenarios for their most important features and functionality.
WHY? Because they want to make sure the product works.

There is one additional test type I want to mention that does not really fit on the group of tests I already described, but since it is very useful I will add it anyway.

8. Smoke Test
WHEN? If you don’t have time, and need to make a fast analysis of the application in order to understand if you should run more tests on a specific area.
WHAT? Even shorter than the Sanity Test, the smoke will include only quick scenarios that will point at major areas of the product. The idea is to probe these areas in search of smoke that will signal there is a FIRE hiding bellow the surface.
WHY? You either don’t have time or you don’t know where to start and you need a quick test to serve as a very high level indicator for your version or product.

I am sure I missed some test types, and I will be happy to include them in my personal list if you care to comment on them, but these are my main and most important test types.

At the end of the day, these types will only serve as a starting template and you will need to create your custom test suites that will suit the needs of your product and project.

Good Luck & Have Fun!

I just finished reading a book that EVERYONE who has anything to do with the software development process needs to read ASAP (and I mean, before you write your next line of code or perform your next tests).

The name of the book is

Why Software SUCKS… and what you can do about it

by David S. Platt

(you can read the first chapter here for free)

First of all, I need to thank my friend Arie for recommending it. And if you go and read it after this article tell me what you thought about it.

The book is not intended for developers or tester (I guess it is, but in a very sutile way), but for users of Software Products; and it explains with a lot of humor and good examples why the software products we write don’t do the things their end-users expect them to do.

(Spoiler Alert: I’m going for the bottom line of the book here, so decide whether to skip this paragraph) The author says, and I agree with him, that the main problems is that (many) programmers write software that they like and not software their customers will like, thinking that “If I design something that I like, the user will surely love it!”, which most times is absolutely wrong!
I specially liked a phrase he repeats many times: “Know Thy User, for He Is Not Thee”, I’m gonna print it and hang it in my office

The book is not long, so it should not take you long to read it. You can also read chapters 1, 2, 4 & 5 quickly and leave the rest for later…

All in all I did enjoy all the book, even though I am less of an MS fan and the author is. I love my Mac as well as my 2 Win PCs, each for their own strengths.

One thing for sure, it gave me a good perspective to understand how each of us needs to go about our jobs. You can say that this is nothing new since we talk about it constantly, but it gave me more practical ideas on how to write and test based on what our users are expecting us to do.

A friend called me yesterday asking for help. He could not find the exact scenario for a critical bug he was trying to report.
We sat and after the third reproduction I started seeing a pattern, within 5 minutes of playing together with the bug we got to the exact reproduction scenario.

This is an example (although not the most common!) of Pair-Testing.

Pair-Testing is the exercise where a pair of testers (or any other pair of people) perform tests TOGETHER (using only one interface with the AUT), with the objective of complementing each other with their backgrounds, points-of-view, experience, knowledge, or even luck. This is one of those examples where “controlled noise” provides synergistic results.

The principle is simple: an additional person that looks at what your are doing and thinks about additional or different operations. (Nothing fancy, but then again most genial things seldom are…)

Surprisingly enough not many testers use (or know about) Pair-Testing regularly, and even less Organizations have it as an encouraged or accepted testing approach. But those who start using it understand it’s strengths and keep using it on those situations where it provides added value to the system.

When can you use Pair-Testing?
- When you think that you’ve tested enough and are not able to detected significant new bugs
- When you feel “tester’s block” and need to get some new energy into your process
- When creating user scenarios on new or existing functionality
- When you need to find important bugs fast (creeping deadline, sudden patch, etc).
- When you are not able to pin-point the root-cause of a bug or problem.

The drawbacks of Pair-Testing:
- It employees the time of 2 people instead of one.
- Not all 2 people can work together like this, sometimes you are not able to coordinate the efforts and communicate correctly with your partner.
- It’s not always trivial when to stop working together, you need to define a good exit criteria.

I suggest you try it next time you run into a situation where an additional pair of eyes might make all the difference…

Going into the subject of Test Project Strategy and more specifically around MTPs there are 2 points where we are not always sure what to do and what are our responsibilities as testers and test managers:

Product Risks & Project Risks

The first step is to understand these are 2 different entities:

Product Risks are areas in the AUT where there is a high risk you will find (important or numerous) defects, usually due to changes or other internal factors.

Project Risks are situations that may or may not happen (risks), if they materialize they usually cause delays in the project’s timelines, and the source of these risks may be internal or external.

Product Risks
As testers one of our tasks is to manage Product Risks.

We are paid (at least in part) to be aware of all Product Risks, to make sure the rest of the project team is also aware of them, and to coordinate this information with Project Management to make sure our schedules are taking these risks into account.
In addition, we are expected to plan our testing strategy based on these risk, scheduling more tests (and earlier tests) on areas with higher risks in order to find these issues faster.

My preferred method for handling Product Risks starts by listing them as part of my MTP and reviewing them with all the Project Stakeholders. During these reviews I try to get inputs into the relevance of these risks, as well as information about additional risks I may be unaware off. Once the project starts we review and update all product risks as part of the coordination meetings with the rest of the project team.

Examples of Product Risks are:
- Complex features affecting multiple areas of the existing product, like an upgrade/migration of the system.
- New Technologies used in the product; for example a new DB server, a new programming language, a new integration, etc.
- New Developers or Development Teams, who may lack experience and thus pose a higher risk to the existing product.
- Tight Schedules, that make people work in a rush and commit more mistakes.
etc

Project Risks
The responsibles for Project Risks are the Project Managers, who are also in charge of the project’s schedule. The QA, as part of the overall team, is responsible for the Project Risks within our work areas.

Project Risks are usually managed in an Excel Sheet (or Google Docs Spreadsheet) where we list and manage all the risks for the project. For each risk we collect and manage the following information:
1. Risk Name & Description
2. Risk Index (we use this field in order to sort our list) – calculated by multiplying its probability by its consequence
3. Risk owner
4. Date of relevance – when does the risk starts been relevant and prevention actions can start taking place
5. Prevention Actions – how to avoid this risk
6. Contingency Plans – what do we do if the risk materializes

The most usual project risks related to the QA & Testing work are:
- Delays in the delivery of the AUT for testing
- Lack of technical knowledge on specific areas of the product
- Lack of testing environments and/or data that effectively simulate real customer usage
etc.

Risks are a big part of any manager’s work. We are expected to be in the lookout for the things that may happen and prepare accordingly.

Most of the task related to Risk Management are not complex but they require good understanding of the project and product as well as the strict discipline required to keep following and managing these risks throughout the whole lifecycle of the project.