– Is the size of a test important?
– How big and comprehensive should a test case be?
– Should tests be always small, or should you write comprehensive tests that cover end-to-end and complex scenarios in “one-go”?
All of these are important questions, and even if there is no text-book answer for them there are some guidelines you can consider when wondering about this subject.
My thoughts on test sizes were prompted by 2 separate events. This week we are starting a long-range project to re-factor PractiTest’s test automation framework (the one we use to test ourselves!). This time we are making a conscious effort to keep our tests as short as possible and to re-use them as part of more complex test-sets.
I am also guiding one of our PractiTest users on his custom deployment, and we are trying to define how deep and extensive should the tests be in order to answer his needs and constraints. With this customer we are reaching the conclusion that most of the tests should be kept small and high level; but still a limited number of regression scenarios will be written very detailed and covering the system from end to end.
When considering the right size and depth of your test the spectrum is extremely wide. On the one hand you can write a word-by-word scripted test, allowing you to take ‘any person from the street’ and use him to test an application he hasn’t ever seen. On the other hand you can give an application to a tester and ask for her “professional opinion” without giving her any indication to what the AUT should do or what you want to check, then come back 3 days later to see what she has to say.
I personally think that both of these extremes are wrong! But there is a large space in the middle to choose from, and many times the right choice will not be easy to find or trivially clear.
I personally prefer my tests to be small and as high-level as possible. I believe that if I take good testers and we give them an introduction on what the AUT should do and who it should serve, then I don’t need to provide them with step by steps cases in order for them to do the job right.
Instead of creating extensive test cases I prefer to user generic check-lists and heuristics to help me go over my application. These tools allow me and my testers to test thoroughly and to find the important stuff; but on the other hand it doesn’t drive us mad by writing kilometric tests with low-level steps that no one really needs and just frustrate testers simply by needing to read them.
There is also an added point to having high-level tests, they force us to think when we test! I’ve seen that if you give even an experienced tester a very detailed test she will turn into a kind of live-test-automation-tool, performing each and every step but being so focused on it that she doesn’t pay attention to the “interesting scenarios” lying right next to her predefined test.
When your tests are high-level steps or even charters or threats (like a couple of interesting articles I read this week about Exploratory Thread-based testing by the Bach Brothers – here are both by James & Jon) you actually motivate your testers to THINK BIG and take more responsibility for the test execution, overall achieving better results and finding more interesting and severe bugs.
On the other hand, having no tests at all and trusting on the intuition of your testers is a sure recipe for failure and important escaping defects; specially when not all your testers are experts in all the system, and even worst if they need to work under pressure when needing to provide results quickly.
So I guess that at least for me size does matter in my tests! I try to keep them small and smart as possible.
What do you think?
Practitest is an end-to-end test management tool, that gives you control of the entire testing process - from manual testing to automated testing and CI.
Designed for testers by testers, PractiTest can be customized to your team's ever-changing needs.
With fast professional and methodological support, you can make the most of your time and release products quickly and successfully to meet your user’s needs.