A good tester asks good questions

Testing a new application or product is always challenging.

There’s the challenge of working with new technologies, new languages and even new platforms; forcing us to investigate and learn how to cope with these new paradigms.  But in a sense all these technical issues can be easily solved by turning to the best-friend of most geeks (like me).

Chances are that if you are asking yourself what tools to use in order to automatically test a new technology, how to write tests for a specific scripting language, or what are the weak spots on an operating system or mobile platform; the answers will already be available on the Internet, written by some early adopter or tester who took on the challenge and posted a paper or blog explaining what he did and how.

But the real challenge is coping with the things that cannot be searched on the Internet, and the ones you will need to figure out by yourself. I am referring to the functional and business aspects of your new application and the way your users will work and interact with your product once it is released.

 

Who do I  talk to in order to get information?

The first question to ask is “Who do you need to talk to in order understand what to test?” and the answer, at least in the beginning of the process, is simple – TALK TO EVERYONE YOU CAN.

When you start a new project or when you’re asked to test a new application you need to start by mapping all the people who can provide you with information.  So make a list of everyone you can talk to – Marketing, Development, Sales, Support, the CEO – and try to talk to each of them in order to understand what you need to test (and why)!!!???

Even if you don’t manage to talk to all of them (sometimes it can be tricky to get a 30 min session with your CEO…) at least try to make the list and cover as many different people as possible in order to get different perspectives and testing inputs.

 

What questions to ask?

Here is the hard part of the problem…

Imagine the following scenario:

You finally schedule the meeting with your CEO in order to get her inputs into what should be tested in the system. You arrive at the meeting and ask her: “Mrs. CEO, I have been given the task of testing the new product, and I wanted to ask you what do you think should be tested?”.
She stops and looks at you for a couple of seconds before answering: “Well, I don’t know, aren’t you the testing expert here…?  I guess you better test everything, right?  We don’t want any bugs slipping out the door, do we?”

What’s wrong with the scenario above?  Well, basically we came to the person and asked him the wrong questions…

If you pay attention to what the CEO said she was right, we are the testing experts.  She expects you to come up with what should be tested in the system, but on the other hand she is also living under the illusion that everything should or even can be tested (something we all know is impossible and even when possible, not economical in the long run).

The problem here is that we asked someone else to do our work for us, to come up with what to test, instead of asking for the input we need to understand by ourselves what needs to be tested and how.

So going back to our CEO meeting scenario, what questions would have been valuable to ask?  Well, you need to ask for her inputs on the system, without even talking about the testing operations.  Try to understand what areas are important from a user perspective, or based on our competitive advantages, or based on what makes our application unique, etc.  You should focus your questions on what is important for them.

Here are a couple of examples you can ask your CEO or VP Marketing or even Director of Sales:
– What are the most important aspects of our Product?  The things that make us unique?
– What areas in the product are the most widely used by our customers?  What type of things would make our users angry and make them choose not to work with our product?
– Where is the market focusing on today?
– How are we better than our competitors?  What areas are the ones that are the most problematic in our competitors, the same areas where we want to exceed?
– Are there any risks you think we should be specially aware off?  Risks in our technology, risks in the product?

Notice that we didn’t ask them what to test, but we did ask what’s important in their eyes (based on their experience).

In the case of the CEO, Marketing and Sales functions we will want to talk about stuff that relates to the functionality of the product.

If we were to talk to our Support Team we would ask them questions related to the areas in which our users find bugs, focusing both on the places where there are the largest amount of bugs as well as where the most critical issues are found.

Finally, when talking to our development peers we will ask them about technological risks, as well as places where they are making the most changes, or where the product is relatively weak or complex and so where we should be putting more testing efforts.

 

The art of listening (and putting together the puzzle)

So what do you do with all the information?  Basically you need to take the stuff you got, and process it in order to get a 360-degree reading of your application.  What do I mean by 360-degrees?  I mean from all the different angles that matter: Technology, Usability, Supportability, Competitiveness, etc.

After all, your work is to test and to provide visibility into whether the application is meeting the quality standards of your users and stakeholders.  The only way to do this is by understanding what’s important to all of them and creating a test plan (or work plan!) that will effectively cover it.

,

  • JHedges

    Joel,
    Great post! When I started my current job five months ago, I spent a lot of time asking everyone – IT, support, training, development, marketing – the question “what do you think the most important part of the application is?” On one hand, I was trying to learn the application I'd be testing. On the other hand, I was trying to find out where the weak spots were in the application, either perceived or real. The areas that were mentioned by more than one group, those became the most important areas to test, followed by the areas currently under development, followed by the areas that training and support had the most questions/problems.

  • Kavitha

    Excellent Post! This is one of the best article.

  • joelmonte

    That was actually the point I was making Jesse.

    Sometimes devs, PM's and other guys in our work-places think that because we are professional testers we have some sort of Automatic-Testing-Risk-Analysis tool that we plug into the code and automatically diagnoses the places where we should tests more vs. less, and even the different scenarios we should be testing… Kind of a testing magic wand!

    Maybe we should add this feature to PractiTest on one of the next releases ;-)

  • joelmonte

    Glad you liked it!

  • halperinko

    Very good post,
    I normally call this stage “Divide & Conquer” – From the QA perspective (one of testers “hat's”) it's not only asking the right stuff, but also verifying that all views are in some level of In-Sync.
    And as early these ambiguities are cleared, the better the product answers the needs of all stake holders.
    So in parallel to learning the needs and priorities, we can give initial feedback to the product and related artifacts which are already there.

    This is true not only to new products, but for every version and every feature in the product.

    Since what matters to each stakeholder, is almost product agnostic – it should be quite easy to maintain a check list of questions to ask each position holder.
    (All CEOs have similar concerns, so does Marketing, Support, Dev …)
    Let's build that check-list together, and improve it with practice.

    Thanks
    Kobi Halperin

  • Vishnudatta

    nice things added.
    thanks

  • Deepaksgawande

    A very good post ever read…but if we have very little time to test then how can we do ask all above questions to the different persons.
    Deepak
    deepaksgawande@gmail.com

  • http://twitter.com/percyholl percyholl

    These are really amazing thoughts that you have shared over here  about google.  Google is very helping tool for every one we can find any type of information from google.
    mobile signal booster

  • Dapatel

    Hi,
    Verifying a product is accessible to the people having disabilities
    (deaf, blind, mentally disabled etc….nice blog for software testers…
    Thanks

  • Pingback: What makes a good tester? | Magnifiant: exploring software testing

  • Pingback: What if – and Does it Matter « Complexity is a matter of perspective