A Good Tester Asks Good Questions
Testing a new application or product is always challenging.
There’s the challenge of working with new technologies, new languages and even new platforms; forcing us to investigate and learn how to cope with these new paradigms. But in a sense all these technical issues can be easily solved by turning to the best-friend of most geeks (like me).
Chances are that if you are asking yourself what tools to use in order to automatically test a new technology, how to write tests for a specific scripting language, or what are the weak spots on an operating system or mobile platform; the answers will already be available on the Internet, written by some early adopter or tester who took on the challenge and posted a paper or blog explaining what he did and how.
But the real challenge is coping with the things that cannot be searched on the Internet, and the ones you will need to figure out by yourself. I am referring to the functional and business aspects of your new application and the way your users will work and interact with your product once it is released.
Who do I talk to in order to get information?
The first question to ask is “Who do you need to talk to in order understand what to test?” and the answer, at least in the beginning of the process, is simple – TALK TO EVERYONE YOU CAN.
When you start a new project or when you’re asked to test a new application you need to start by mapping all the people who can provide you with information. So make a list of everyone you can talk to – Marketing, Development, Sales, Support, the CEO – and try to talk to each of them in order to understand what you need to test (and why)!!!???
Even if you don’t manage to talk to all of them (sometimes it can be tricky to get a 30 min session with your CEO…) at least try to make the list and cover as many different people as possible in order to get different perspectives and testing inputs.
What questions to ask?
Here is the hard part of the problem…
Imagine the following scenario
You finally schedule the meeting with your CEO in order to get her inputs into what should be tested in the system. You arrive at the meeting and ask her: “Mrs. CEO, I have been given the task of testing the new product, and I wanted to ask you what do you think should be tested?”. She stops and looks at you for a couple of seconds before answering: “Well, I don’t know, aren’t you the testing expert here…? I guess you better test everything, right? We don’t want any bugs slipping out the door, do we?”
What’s wrong with the scenario above? Well, basically we came to the person and asked him the wrong questions…
If you pay attention to what the CEO said she was right, we are the testing experts. She expects you to come up with what should be tested in the system, but on the other hand she is also living under the illusion that everything should or even can be tested (something we all know is impossible and even when possible, not economical in the long run).
The problem here is that we asked someone else to do our work for us, to come up with what to test, instead of asking for the input we need to understand by ourselves what needs to be tested and how.
So going back to our CEO meeting scenario, what questions would have been valuable to ask? Well, you need to ask for her inputs on the system, without even talking about the testing operations. Try to understand what areas are important from a user perspective, or based on our competitive advantages, or based on what makes our application unique, etc. You should focus your questions on what is important for them.
Here are a couple of examples you can ask your CEO or VP Marketing or even Director of Sales:
- What are the most important aspects of our Product? The things that make us unique?
- What areas in the product are the most widely used by our customers? What type of things would make our users angry and make them choose not to work with our product?
- Where is the market focusing on today?
- How are we better than our competitors? What areas are the ones that are the most problematic in our competitors, the same areas where we want to exceed?
- Are there any risks you think we should be specially aware off? Risks in our technology, risks in the product?
Notice that we didn’t ask them what to test, but we did ask what’s important in their eyes (based on their experience).
In the case of the CEO, Marketing and Sales functions we will want to talk about stuff that relates to the functionality of the product.
If we were to talk to our Support Team we would ask them questions related to the areas in which our users find bugs, focusing both on the places where there are the largest amount of bugs as well as where the most critical issues are found.
Finally, when talking to our development peers we will ask them about technological risks, as well as places where they are making the most changes, or where the product is relatively weak or complex and so where we should be putting more testing efforts.
The art of listening (and putting together the puzzle)
So what do you do with all the information? Basically you need to take the stuff you got, and process it in order to get a 360-degree reading of your application. What do I mean by 360-degrees? I mean from all the different angles that matter: Technology, Usability, Supportability, Competitiveness, etc.
After all, your work is to test and to provide visibility into whether the application is meeting the quality standards of your users and stakeholders. The only way to do this is by understanding what’s important to all of them and creating a test plan (or work plan!) that will effectively cover it.
