Teaching programmers to test

Would you trust a programmer to test your application?  It’s like asking a Fox to guard the chicken-house, right?

Developer testing - fox

Well, sometimes you don’t have a choice, or you do but that choice is to release the application untested…

As part of an Agile testing consulting engagement I am doing at a friend’s company, yesterday I started providing short training sessions for programmers who need to learn how to test better.  It’s not that this company doesn’t have testers, they have good testers!  But as many other agile teams they have much more testing tasks than available testers, and they want programmers to take part in at least some of their testing tasks.

Programmers are not good at testing!

A couple of months ago I wrote a post explaining why I think programmers make poor testers, and I still think that in a sense it is like asking a dog to fly (take a look at the cool picture in the post 😉 ). On the other hand I also believe that with some good intentions, hard work, and perseverance you can teach a dog to skip far enough and that way to jump over large obstacles.

In the same way, you will not be able to magically transform programmers into a good testers overnight, but you can start by working with them on their weaknesses and then teaching them some simple and effective testing techniques.  With good intentions, a little practice, some hand-holding, and constant feedback you will be able to get some good testing help in your project.

Step 1 – understand the limitation and weaknesses of a programmer

I started my session yesterday by going over the points I listed in my earlier blog that make programmers “less-than-perfect-testers”:

– Parental feeling towards their own code.
– Placing the focus mainly on positive scenarios, instead of actively looking for the bugs.
– Tendency to look at a “complex problem” as a collection of “smaller, simpler, and isolated cases”.
– Less End-to-End or User-Perspective oriented.
– Less experience and knowledge of the common bugs and application pitfalls.

It made a big difference during the session to not only list the weaknesses but talk about why each of them is naturally present in programmers due to the nature of their work, their knowledge and training.

Step 2 – how to plan tests

I’ve seen that many (most?) programmers have the perception that testing requires little or no planning.

Maybe we testers are guilty of this misconception since we don’t get them involved in our test planning process as much as we should, but the truth of the matter is that when we ask them to test they they automatically grab a mouse and start pressing on the buttons without much consideration or thought.

Test Planning is a cardinal aspect of good tested, so I gave them a couple of ideas and principles planing:

When as a team they are asked to test, they should make sure to divide the tasks so that each of them is testing the code developed by other programmers as much as possible.

2.  Work with your testing team to create test sets.
This specific team is using PractiTest (the hosted QA and Test Management platform my company is developing), but it can be any other tool or format (even word and excel!).  They should sit with their testers and define what test cases need to be run, reusing the testing scripts already available in their repositories.

3.  Expand your scenarios by making “Testing Lists”

– Features that were created or modified (directly or indirectly)
– User profiles and scenarios to be verified.
– Different environments / configurations / datasets to be tested

The use of these lists is two-fold.
They help you get a better idea of what you want to test while you are running your manual test scripts, and they also serve as a verification list to consult towards the end of the tests when you are looking for additional ideas ow when you want to make sure you are not missing anything of importance.

4.  Testing Heuristics – SFDEPOT (read San Francisco Depot)
The use of (good) heuristics greatly improve the quality of your testing.
I provided the programmers with the heuristic I learned fist and still helps me up to this day.  I read about SFDEPO(T) from James Bach some years ago – you can check one of the sources for it from Jame’s Site.

SFDEPOT stands for:
Structure (what the product is)
Function (what the produce does)
Data (what it processes)
Platform (what it depends upon)
Operations (how will it be used)
Time (when will it be used)

There are other heuristics and Mnemonics you can take from the Internet…

Step 3 – what to do when running tests

We talked a lot about tips to help perform good testing sessions.

1. Have a notebook handy.
In it you can take notes such as testing ideas you will want to test later, bugs you ran into and want to report later in order to “not to cut your testing thoughts”, etc.

2.  Work with extreme data.
Big files vs. Small files
Equivalent input classes [ -10 ; 0 ; 1 ; 10,000,000 ; 0.5 ; not-a-number ]
Dates: Yesterday, now, 10 years from now

3.  Think about negative scenarios.
– How would Mr. Bean use your software?
– How would a hacker try to exploit the system?
– What would happen if…? (blackout, run out of space, exceptions, etc)
– What if your  2 year old would hijack the keyboard in the middle of an operation?
– etcDeveloper testing - Mr bean

4.  Focus & Defocus.
(Thanks Ayal for teaching me this principle this year!)
I used this technique to explain to them that during their testing process they need to make a conscious effort to always look at the bigger picture (application, system, process) and not only focus on the specific function they were testing.

5.  Fight Inattentional Blindness.
I used the following film of the kids passing the balls to explain the concept of Inattentional Blindness and it worked great!
We were 8 people in the room, and only I had seen the video before.  Out of the 7 participants one is currently a tester, another one is a former tester turned programmer, the rest are “regular programmers”.
The cool thing is that only the 2 testers saw the Gorilla the first time…  talk about making a point!

Step 4 – what to do when (you think) you are done testing

We talked about how even when you think you are done testing you should always make sure there is nothing else that should be tested, and what techniques they can use in order to find these additional places:

1.  Walking the dog
Taking a break from your tests, doing another tasks for 30 – 60 minutes, and then returning to review the tests you did and what you found.  This break usually helps to refresh the mind and to come up with more ideas.

2.  Doing a walk-through session to review the tests you did.
Most of the time you will be able to get more ideas as you explain to your peers about tests you just did.
The funny part is that many of these ideas will come from yourself and the things you think about when you are explaining your tests to others out loud.

3.  Ask for ideas from other developers or testers.
Simple as it sounds, come to others in your team and ask them to pitch you ideas of stuff they think you could test.  90% of the stuff you will already have tested, but the other 10% might prove useful too!

In the end is a questions of mindset and motivation

One of the things I like most of agile teams (and many non-agile but yes smart development teams) is that they define Quality to be the responsibility of the whole team and not only of the testing guys running the tests tasks at the end of the process.

My final piece of advice to the group was that everything starts from them, and their understanding that testing is not a trivial task and definitely not a reprimand for doing a bad job or for finishing their tasks ahead of time.  What’s more, I am sure that once these guys start testing better and gaining a testing perspective of their application they will also start developing better software too.

Now I have another 3 development teams I need to train.
Let’s see how it goes with them 🙂

About PractiTest

Practitest is an end-to-end test management tool, that gives you control of the entire testing process - from manual testing to automated testing and CI.

Designed for testers by testers, PractiTest can be customized to your team's ever-changing needs.

With fast professional and methodological support, you can make the most of your time and release products quickly and successfully to meet your user’s needs.

, ,

7 Responses to Teaching programmers to test

  1. Joe Strazzere December 23, 2010 at 1:01 pm #

    Excellent article, Joel!

    Please do come back and post a follow-up, once the development teams you have trained do some testing. It would be great to hear what worked well, and what didn't work as well in real practice.

    I've only had the opportunity to quickly train individual developers for short testing roles. The one area that proved to be the most effective by far was “Think about negative scenarios”.

    And I have always shared a set of slides with each developer as a way to break the ice and start things off. http://strazzere.blogspot.com/

  2. joelmonte December 23, 2010 at 1:19 pm #

    Thanks Joe and great set of slides, I may use them in my next sessions if you let me…

    Some of the stuff in there was brilliant:
    “If Developers are Yin, Testers are the Yang”
    “Everything has flaws and given enough time I can prove it”
    “We can't spell BUGS without U”

    and the one that made laugh the most was the dialogue:
    “It works in my machine” -> “Perhaps your machine is the only one where it works”

    I will try to post a follow-up in a couple of months to see what worked and what didn't.


  3. Joe Strazzere December 23, 2010 at 2:03 pm #

    Glad you enjoyed the slides.

    As long as the slides are kept intact (with my name and blog included), anyone is free to use them at any time.

  4. Dayong Fu April 22, 2011 at 4:25 pm #

    I only walked through the article very quickly. I know the point is to teach programmers to test from a tester perspective. But I really don’t like and disagree “programmers are not good at testing!” in the article. Maybe a developer can’t switch to a very good tester directly just because he/she does not have the full knowledge about all aspects of testing domain, but good developer does require good testing skill and thinking. Developer needs to think whether your design and implementation are testable and needs to be responsible for unit testing (part of testing also.)

    – Parental feeling towards their own code. You know what, good parents sometime are much stricter to their children than others.

    – Placing the focus mainly on positive scenarios, instead of actively looking for the bugs. Totally don’t agree, actually good developers spend a lot of effort and thinking to handle the corner case otherwise how we can write robust codes.

    – Tendency to look at a “complex problem” as a collection of “smaller, simpler, and isolated cases”. I think nothing wrong with this. It’s kind of thinking logic or problem analysis skill. Bring complex thing into simple steps and resolve them. This does not mean developer can’t compose some complex scenarios.

    – Less End-to-End or User-Perspective oriented. It depends on your responsibility and scope. I think when a developer becomes senior and senior, his/her responsibility and scope will become much ‘higher’ level or system level, which means he/she need to more focus on end-to-end scenarios and then can design the system better. That also means good developers do care about end-to-end scenarios from user perspective, other how can design the whole system better.

    – Less experience and knowledge of the common bugs and application pitfalls.

  5. joelmonte April 27, 2011 at 5:44 am #

    Thanks for the inputs!!!

    I completely respect your points and I welcome the diversity of your views, still in my experience all the points you list above have not been present in most of the great developers I've worked with.

    I don't mean to say at any point that developers are lazy or that they don't see the importance of testing, just this last week one of the best developers in PractiTest decided (out of his own accord and priority) to spend his time revamping the functional testing platform…

    So in principle we both agree that developers (or at least good developers!) will see the importance of testing. The point where we disagree, is where I think that developers (and even I myself when I work on development tasks fall into this category too) can be blinded by the multiple aspects I named above and thus their testing will be less effective than the one of a tester.

    BTW, this is the reason why I wrote this post, to explain that developers can test as long as we help them overcome their blind-spots.

    Again, thanks for your comments and for bringing forward your points and discussion. I am sure they will provide additional thinking point to the rest of the readers.


  1. Why can’t developers be good testers? | QA Intelligence - a QABlog - December 24, 2010

    […] to write the blog of pairing developers and testers (yet!) but I did write a post about sessions teaching programmers how to overcome their testing limitations (listed above) and giving them some techniques to help them test better.   If you thought this […]

  2. URL - December 13, 2011

    … [Trackback]…

    […] Read More here: https://qablog.practitest.com/2010/12/teaching-programmers-to-test/ […]…

Leave a Reply