Don’t write a single test! Until you know how to do it

We’ve all heard about the “Infinite Monkey Theorem” whereby a monkey hitting keys on a keyboard (typewriter) at random will eventually come up even with the complete works of Shakespeare.

Monkey Typing

Monkey Testing? 🙂

But the problem is that it would take it an infinite number of years, and maybe more importantly the work would be buried under so much junk and gibberish that it would be impossible to find it.

What’s the status of your test cases?

Now take a look at your test repository.  What do you see?
Is there anything in common with the work of the monkey above?

I am not implying your team is made up of chimps typing at random – even though if it is, please take a picture of them and send it back to me, I promise to publish it!!!

But something I see a lot in the context of my work with PractiTest‘s customers is that people tend to concentrate on the quantity of their test cases, and fail to put enough efforts on the quality of their resources.

The result is repository with a lot more test cases than it should actually have, many of them covering the same feature too many times, others describing functionality that was modified a number of releases back, and some that have not been run in a number of years because they are not really important anymore.

I don’t think this comes from incompetence, but I do believe that a big factor for this is the fact that it is easier to create a new test than to find an existing case (or cases) and modifying it accordingly.

Another cause is the fact that it is a lot easier to measure the number of tests than the measure the quality of your testing coverage (and the quality of the individual tests cases themselves).

Process and rules of thumb for writing test cases

A good way of stopping problems of this type is to have some process and rules of thumb in place to help testers write better cases.

ID-10051915Some examples for rules of thumb you can define with the team can be the following:

  • Setting upper and lower limits for the number of steps per test case.
  • Setting maximum number of test cases per feature or functional area.
  • Working with modular test cases that you can use as building blocks for your complex test scenarios.
  • Have separate test cases for positive and negative scenarios.
  • Use attachments and spreadsheets to decouple testing data from testing processes.

Regarding process, this is a little harder but it is also a lot more effective in the long run.  Some examples might be:

  • Before starting to write test cases have a small team create the test list and their scope, only then fill out the steps.
  • Break your test repository into areas, assign each area to a tester in your team and make him/her responsible for all aspects of its maintenance.
  • Have peer test creating or review sessions.
  • Visit and validate “old” test cases & tests that have not run in the last 6 months.
  • Review tests that do not find any issues in the last year.

Create visibility and accountability into your test management system

A big factor that will help or hamper the way you maintain your test cases is how you manage and organize them.  You can obviously do this in your file system repository or using something like Dropbox to share these resources with your team.

But  after your tests grow in size or your team expands above a certain level it makes more sense to find a professional test management system that will help you to organize your test cases and generate good visibility and control over them.

I don’t want to make this a marketing post, but I do recommend you take a look at PractiTest and the way we provide excellent visibility into your test cases with the use of our exclusive Hierarchical Filtering system.

Other than creating visibility, make sure there is accountability for each area of your testing repository.  As I wrote above, it is important to assign testing areas and their cases to specific players in your team.

Give them tasks (and the time) to update and maintain their tests, both during the preparation stages but also during the regular testing process.  You should not expect people to maintain their test cases during the frenzy and craziness of your regular testing cycle.

How do you do it?

ID-100288588Does your team have other process or rules of thumb in place to maintain your test cases?

Share them with us to understand what other ways are there to keep our test cases sane and effective.

, ,

  • Kobi Halperin

    And of course – put the right tools + encouraging instructions to update test cases On-The-Go while performing the tests.
    I found using ALM tools make this easier than trying to update MS-Word copies,
    But many improvement can still be done in these tools – to make these live changes even easier.
    Personally – I don’t like Data decoupling by using external tools – I’d rather prefer having the data presented clearly in similar manner within the tool itself,
    And allowing N-wise reduction abilities.
    A major issue can be the presentation of these in a manner that will allow later review of their coverage “10,000 feet” view with coloring visualizations…

    Also – not quite sure as to the Max limits.

    I think a major part of the problem is that we tend to write test cases suitable for 1st release – which as expected is quite extensive and atomic, thus taking lot’s of time to elaborate – But then we have no more time for converting these into Regression suitable test cases.
    I’d rather leave 1st version cases in high level (as we can’t really predict all their details which are bound to change during implementation), and leave more time to End to End – Regression useful test cases with which we will continue from there on.

    @halperinko – Kobi Halperin

  • Ory Zaidenvorm

    Kobi’s point above in regards to being able to edit test cases easily during execution cannot be stressed enough. I’ve canceled evaluations of test management systems on this alone in the past. Also same as Kobi I also like to build a lean and mean regression suite from the complete bank of test cases most of which can be archived after a they were used to validate highly specified functionality.
    Test review is critical and should be reflected in the test workflows either configured in a tool or set in a process. Similarly having a process for test cases that need maintenance is highly valuable. Implementing other processes around regular reporting of test case status is also paramount. First of all because if you’re not reporting on them regularly -what’s the point of having them? And secondly the reporting process itself if done correctly can highlight problematic areas in your test repository.

  • Kimberly Loewen

    I agree. You must plan your test cases to get quality testing. Be strategic- look at the requirements and prioritize. Another good strategy I have used is don’t just think about one individual test cases, but think of the integration points and end to end testing as well.

  • joelmonte

    Thanks for the input Kobi!

    I agree with the absolute need of re-writing test cases once you switch them into regression, but on the other hand I like the approach where testers edit the test on the run in order to make this adaptation.

  • joelmonte

    Echo that Ory, specially the part about regularly reviewing and maintaining your test cases.

    A very big sin of testers is thinking that tests do not need regular review and maintenance – just like code! – and we end up having tons of rubbish in our database that renders big parts of our system useless.

  • joelmonte

    Agreed!

    You need to look at the system and your tests from multiple angles and approaches.

Shares