PractiTest QA Blog

The other day on QAForums, Mark Crowther posted an interesting question:

“If I asked “What Test Types sit under Non-Functional Testing?” you might reply with Accessibility, Compatibility, Performance, Load, Stress, Recovery and a bunch more.
So what sits under Functional (Testing)?”

His question is not trivial.
Maybe it is because we tend to think of Functional Testing simply as making sure the application does what it’s supposed to (and doesn’t do what is not supposed to). Or maybe because many times we confuse testing types with techniques and think that Equivalent Partitioning and Border Case Testing are examples of testing types. In either case, it is something that got me (and a couple of other QAF’s participants) thinking…

Playing the Devil’s Advocate I will start by questioning whether we should care about Functional Testing Types or Categories? Maybe Mark is simply wasting our time with this subject?

The answer is that Mark is not wasting anybody’s time, he is bringing forward a good point. If we don’t have a proper list of Functional Testing Types in hand when planning and/or executing our testing operations, how can we make sure we did not miss something important?

Think of it as a supermarket checklist, you don’t want to get home and then remember that you forgot to buy something important. That’s why you make a list and go over it during the shopping and most importantly before leaving the store.

If we agree that we need a defined list of test types, then what conforms such a list?
I’m sure we all work with multiple types and the problem may be that we call them by different names, so I will leave aside naming conventions and write down my personal Functional Testing Type List.

1. Function-based testing
This is the most straight forward test, where we take each piece of the application and test it individually. For example you can take a date field and try entering all the legal and illegal value types to asses its behavior as a standalone component.

2. Scenario-based testing.
Where we take end-to-end user scenarios and run them in order to review the behavior of the system. In this category or type it is important to work with simple as well as complex scenarios, thinking about all possible real-world situations.

3. Negative testing.
Looking at all the possible things that a user may do wrong in the system and making sure that a correct indication is displayed and no permanent damage is done to the system or its data.

4. Long-range testing.
These are all the scenarios where the system needs to work correctly over long periods of times, including all sorts of maintenance activities that take place only at certain times in the application life-cycle (e.g. log-file recycling).

By now I’m sure you noticed there is some overlapping between the different test types. If you think about it, this also happens in the non-functional test types; but most important is the fact that our objective is to assure we have the correct application coverage by combining (and sometimes even mixing!) all the Functional Test types above.

This list is only my personal list, based solely on my own experience.
If you have additional types add them in the comments section and help me (as well as the rest of the readers) to learn from your experience.

Quoting James Bach: “The more likely the problem is to happen, and the more impact it will have it if happens, the higher the risk associated with that problem”
Or in simply put:

Risk = Impact * Likelihood

If risk-based testing is so simple then how come most projects fail to implement it?
The answer lies in multiple factors but I think that it boils down to lack of discipline and of a good and structured test planning methodology.

Whatever your reasons (excuses) there are 3 things you can do that will increase your project execution efficiency.

1. Perform a Risk Analysis
In many places the most important thing missing is the activity of sitting down and analyzing your project and product risks. Remember that these risks change for each project, so even if you can use a previous analysis as a starting point you cannot use it as your guideline.
Remember to look at all your risk factors (e.g. Functional, Technological, Environmental, etc) and to include in your process stakeholders from other groups (Dev, PM, Support, etc) to provide inputs based on their knowledge and experience.

2. Use risk-based inputs as part of your scheduling criteria
You schedule your tests based on many factors, but remember to use your risk analysis as one of these factors.
You can additionally use your risk analysis in order to negotiate with Development the internal release schedule by which they will provide their features to the testing team. This is specially effective if you had a representative from Development Team when you executed your analysis.

3. Assign a risk based priority index to your test
Projects are always late and you will always be asked to cut down from your original schedule. If you assign each one of your test a risk index you will be able to:
(a) Know where to cut faster.
and
(b) Show the rest of your stakeholders where you need to stop cutting and where the schedule needs to be prolonged due to risk concerns.

It is important to remember that your understanding of the Product and Project Risks will change as your work progresses. You need to review your risk assessment multiple times during project and to update your working assumptions accordingly.

Risk-based testing is an ongoing process, it is also only a tool to help you focus your work and make the most intelligent decisions throughout the project. Like all other tools, it can do the job right or wrong, it will mostly depend on the person who uses it wisely.

I had an interesting conversation with an R&D Manager who was telling me that the reason they worked with MS-Excel instead of using a Bug Tracking Platform was because she didn’t want to waste the time needed to install and configure such as system. Her rational was the Excel was already there, required no installation and all the Engineers knew how to use it.

At first I thought she was kidding but as soon as I realized it was for real I asked for 10 minutes in order to show her why her incomplete time equations where driving her to the wrong conclusion.

I went to the her office’s whiteboard and together we created the following table:

To make a long story short, using her project numbers and expected statistics I showed her that while the setup costs are real, the savings in time needed to report, edit and manage the bugs would more than pay for it, saving at least 2 man-days or about 37% of time invested in bug operations.

I also pointed to the fact that if she were to choose a hosted system that doesn’t require any installation or configuration (such as QuackTest ) she may be able to save an additional 4 hours of work.

The numbers I used on the table where intentionally conservative, as I think most of us would agree that searching for bugs in an excel can take more than 1.5 minutes, and I also left out the time wasted while the excel file is locked and/or even worst if it gets corrupted by the multiple savings and updates from different clients and machines. I personally feel that even on a small project working with an organized system can save closer to 75% of the time invested in Bug Management & Tracking.

At the end of the day the R&D Manager got convinced by our reasoning, hopefully one more Company that will start treating bugs and their management with more caution and respect…