“Do tester need to be as technical as programmer
to be successful at their jobs?”

I got plenty of answers, so I will only post some of them representing the main opinions threads:

@TestAndAnalysis - Testing is a technical discipline that is different to programming and testers add a lot of value to projects
@huibschoots – No! It depends on the context they work in. But in general they need to have some basic technical skills (or the will to learn)
@datoon83 – I’d say that all people involved in the delivery need to talk 1 language – that of the domain and customers!
@klyr – Technical and non-technical testers have a different approach to testing, and will find different bugs.
@sgershon – Certainly do. Not sure about “more/less technical” as programmers, possibly they have to be “differently technical” than them.
@halperinko – @sgershon @joelmonte I normally look at it as in depth knowledge for devs, vs. System-wise knowledge for testers. (Some don’t follow rules)

There were also a couple of tweeps who replied with blog posts of their own sharing their opinion on the subject. 
@diamontip – my answer – do tester need to be as technical as programmers to be successful at their jobs? bit.ly/v5vcX
and
@adampknight – I personally dislike calling testers “technical” wrote about it here bit.ly/tVHDOB

To all the tweeps above and those I missed, thanks for the great feedback!

But as you surely guessed I have my own opinion on the subject and I want to share it with you, so here it goes…

My definition of Technical

Specially after reading Adam’s post I feel the need to explain what do I mean by technical, or better yet how do I differentiate a Technical Tester from a Non-Technical Tester. (If you read my previous blogs on “Why are some tester not really Professional Testers” then you should already have an idea…)

A Technical Tester is not afraid of doing most of the following stuff on a regular basis as part of his job (without any specific order):

- Understand the architecture of the product he is testing,
including the pros & cons of the specific design, as well as the risks linked to each of the components and interfaces in the product.

He then uses this information to plan his testing strategy, to execute his tests and find the hidden issues, and also to provide visibility to his team regarding the risks involved in developing a specific feature or making a given change to the system.

- Review the code he needs to test.
He can do this on a number of levels, starting from going only over the names of the files that were changed, and all the way to reviewing the code itself. This information will provide valuable inputs to help decide what needs to be tested and how, as well as to find things about the changes that might have been missed by the developer or the documentation.

BTW, by code I mean SQL queries, scripts, configuration files, etc.

- Work with scripts & tools to help his work.
A technical tester should be able to create (or at least “play”) with scripts to help him run repetitive tests such as sanity or smoke, and tasks such as configurations, installation, setups, etc.

He should also be able to work with free automation tools such as Selenium or WATIR (or any of the paid ones like QTP, SeeTest, TestComplete, etc) to create and run test scripts that will increase the stability of the product in development, and over time save time…

- Be up to date with the technical aspects of his infrastructure
(e.g. browsers, databases, languages, etc)
He should read the latest updates on all aspects of his infrastructure that may have an effect on his work. For example new updates to his O/S matrix, known issues with the browsers supported by his product, updates to external products they integrate, etc.

With the help of Google alerts and by subscribing to a couple of newsletters anyone can do this by reading 5 to 10 email 2 or 3 times a week. The value gained from becoming an independent source of knowledge greatly exceeds the time invested in the efforts.

- Is able to troubleshoot issues from Logs or other System Feeds.
He is aware of all the logs and feeds available in his system, and uses them to investigate more about any issue or strange behavior.

This information is helpful during testing to provide more information than simply writing “there is a bug with functionality X”. And it will be critical if he is called to work on a customer bug, where he needs to understand complex issues quickly and without access to all the information.

In addition to the above, a technical tester should also be able to:
- Provide feedback and run the unit tests created by his programmer peers.
- Run SQL Queries on the DB directly to help verify his testing results.
- Install and configure the system he is testing.
etc.

Sounds like Superman or MacGyver?

It may sound like this, but actually it’s not!

As testers we work on projects that revolve around Software, Hardware, and/or Embedded products. The only way to do a good job in testing them is to have a deep understanding of both angles: technical and functional.

This doesn’t mean that you need replace or have the same technical dept as your developers, or surpass your Product Marketing’s knowledge of your users.

You need to achieve a balance, where you have “enough” knowledge and understanding of both these areas in order to do your job as a tester, or rephrasing the tweet from @halperinko – as testers we should achieve system-wide knowledge, as opposed to the in-dept knowledge required from the developers or the product marketing guys.

Is it black and white?

This time quoting “Cokolwiek” who commented on one of my latest posts, “Not everything is black and white”. Meaning there is no standard to define how technical should a tester be on every project and product.

Like in many other situations, the best answer to how technical do you need to be is: “It Depends…”

You should be at least technical enough to do your job effectively and to talk the same language with the rest of your programming and testing peers.

What do I mean by that?
If you work on a software development firm then you should understand enough of the languages used by your developers to be able to read the code and understand their changes. If you work on a heavily DB-related project then you need to understand enough of SQL and database management. If you work on a Website development firm then you should know enough CSS, HTML and JS, and so it goes…

So if I am not Technical enough, should I quit testing???

Definitely not!
If you like testing and you are good at it, why should you quit? On the other hand, this is a great opportunity to improve your work and (as @diamontip wrote on his blog) increase your market value as a tester

And the best part of it is… it’s not very hard to become a more technical tester! Just start by asking questions, searching the web, reading books, etc.

I’m also confident that if you show the potential to increase the value of your current work to your manager, he won’t mind you investing sometime during your day to learn these new trades (as long as you manage to explain why this is also good for him!).

So, stop making excuses for not been technical enough. Just make a decision (or a New Year’s resolution…) to start working on improving your technical skills!

Go ahead and get rid of the NON-Technical Tester in you!

It will be worth your time and make your job more interesting and satisfying.

What do you think?

- Are there any negative sides to being a more technical tester?
- Maybe there are advantages for testers who used to be developers in the past?

Come and share your opinion or experience on this subject!

Classic examples are:
- When you run a short test on a piece of code that is “under development”, and the programmer just asked you for feedback.
- When you are running the sanity suite for the 900th time on the same system before or after it has been deployed to production.
- When you are checking your own code, just to make sure it didn’t break anything you didn’t intend to break in the first place.
And there are thousands of other occasions when we just say “the heck with it, I can just run a quick test without planning it”.

Well, if you are also like me, as soon as you catch yourself saying this nonsense you take out a piece of paper and write down the 5 or 6 things you want to test…

Why?
- Because when you are testing something you have never tested before you want to make sure you talk to the programmer and understand what the feature should do (and what it shouldn’t)!
- Because specially when you are running the same test over and over again in auto-pilot, you will get distracted and miss something that may be important in your sanity before or after release
- Because when you are checking yourself you are blinded by your own mistakes, and writing down your testing charter is a good technique to get some perspective and catch some of your own bugs.

But maybe most importantly, because you are a professional tester, and even if you are running a short testing session it is important to define your testing scope, and to be able to assess whether your are really done with your tests or if there are other things you should be testing.

If you want to be treated like a professional tester,
the first step is to treat your testing professionally.

When you are a true professional, you see failure
as an opportunity to become a better professional.

Here’s a real life scenario, let’s see how many of you can relate to it:

It is the middle of the day on a regular Tuesday afternoon.  You are the QA Manager of a company developing an Enterprise Application, and last week your team released a minor version that included the fixes from all the patches of the last two months plus four minor features that were requested by product marketing in order to “close some pretty important deals”.  

All of a sudden the phone rings and it is your R&D director “inviting” you to an urgent meeting in his room.

You arrive to find there the R&D director, together with his development team managers, the product marketing manager, and the support team leader in charge of your product who is standing next to the whiteboard…  

As you sit down the support team leader tells all of you about an urgent showstopper that was released in last week’s version and that is expected to affect about a third of the companies who install this upgrade.

I want to stop this scenario (that happened to me about 7 years ago) to ask you a simple question:

What would you and your team do in this situation?

I believe that no two teams would react in the same way, and I don’t want to come up with best & worst case scenarios; but here are two contradictory approaches to serve as points in the possible behavioral continuum.

Scenario No. 1 – Blame & Panic

Step 1
The meeting turns into a witch-hunt where development blames testing for not finding the bug, then testing blames development for not documenting all the changes made in the system, then development and testing blame product marketing for pushing the teams to release even though not all the tests had been completed, etc…

Step2
After the meeting dissolves without any clear action items, support starts telling customers not to install the new release, the programmers start working on a solution without fully understanding the problem, and you are left on the side wondering how did you miss this bug and trying to find the person who should be responsible for it.

Step 3
Since the developers think this is absolutely urgent they decide to send the fix directly to support, and only in parallel they send it to your team for validation and verification.
They do this at 8:30 PM when no one is left in the office and you can only start testing it at 8:30 AM the next day.  About 30 minutes in to your testing cycle you start finding bugs in their new version.
The problem is that your support team already started delivering this solution to the initial set of customer who already complained about the bug.

Step 4
By mid day your team finished the tests on the system, they found that the initial fix only works on about half the supported configurations, and more important it also causes a regression bug on an area not directly related with the fix.
Within 30 minutes you get a new version that is verified and released to the support team by 4:00 PM.

Step 5
Customer support wants to kill both your testing team as well as the developers because now they need to find every company that downloaded the first fix and call to ask them not to install it, or even worst to install yet another fix on top of it.
Product Marketing is also mad at you since they already started getting calls from customers, account managers, and even some of your company’s top executives, all complaining about the mess and bad publicity this fiasco is already creating in the field.  They let you know that as a result of it your company will need to offer large discounts to all customers that complaint, and they think this issue may cause a number of important deals to be delayed or lost…

Step 6

(Step into your time-machine and fast-forward 3-4 months ahead)

Everything is still the same, no one was fired by the fiasco but the atmosphere was tense for about one week afterwards; after that it became water under the bridge.

Your team is about to release another minor version including all the patches of the last months, plus another three features needed for important deals.
As always product marketing is pushing for the release to go out on schedule even though your team got the final built a week late and you learned only yesterday that they included another feature that you were not even aware off…

As they say…

Scenario No. 2 – Solve, Learn & Improve

Step 0 - Don’t Panic!

The last thing you want to do is start looking for someone to blame.  Chances are that more than one person is “to blame” for making mistakes that lead to the issue been released, It is almost certain no one did this on purpose, and most importantly it will not help you to solve the issue!

So try stoping your basic instinct to blame someone else for the issue.

If another member of the team starts the blaming game, immediately ask him how is this contributing to solving the issue any faster?  You can also state that there will be enough time, after the issue is solved, to understand what went wrong and why.

Step 1 – Fix & Test
OK, so there is a bug out there and you better fix it quickly!
Put together the best team you can that will (1) analyze the issue, (2) define the quickest, safest and most effective fix, (3) create this fix, and finally (4) recommend how to test it.
What testing approach to take is not a simple decision either, depending on the bug and the product you are working you may choose to deliver the fix directly without doing any tests and verify only after the issue has been solved (for example if your application is web-based and the servers are down, it is better to get them back up and test once they are up).  On other occasions you may choose to run some or all possible tests in-house before releasing the fix, this is usually the case if the bug is not really critical and the fix may cause bigger bugs such as data-loss or business disruption.

In short, the first step is to create the best solution and find the most appropriate approach to get it to your customers.

Step 2 – Analyze
Once the critical part is over and the issue has been solved, but before people “move on” with their lives and tasks, its better to make sure you understand what went wrong.
The analysis process should never be a witch-hunt, it should be an opportunity where everybody feels safe to collaborate and bring forward all the factors that contributed to the problem happening in the first place (both internal as well as external factors).
This activity is fairly common and it is called a retrospective or post-mortem.

Step 3 – Corrective actions
Once the factors and issues have been identified as part of the post-mortem, the next step is to define corrective actions to prevent this from happening again.
Make sure these actions are clearly defined and actionable (duh!).
Many times we see stuff like “make sure communication is better” but this is not actionable at all, and it will not help anybody to change the way they have communicated up to now.
So as dumb as it may sound, make sure your corrective actions are actionable and they will lead to a change in the way things were been done in your company up to now.

Strong team share their failures while weak teams sweep them under the rug

I remember coming up with a phrase a couple of years ago as part of an presentation I did for a group of testers: “When you are a true professional, you see failure as an oportunity to become a better professional“.

I have seen many development teams where they perform retrospectives but then they are affraid or ashamed to share their results with the rest of the Company.  Would you blame the team for been self-centered and egoists?  I would actually blame their companies for not making sure their work atmosphere encourages teams to take risks and learn from their failures…

You need to make sure your team, and preferably your Company, encourages the sharing of retrospectives and corrective actions.  It is one of the best sources of free advice available.

Also, if you look closely, teams with the confidence and maturity required to openly share their risks and failures are also the most fun and challenging teams to work in.

And if all this was not enough to convince you, just go a head and read the Dilbert Comic Strip published earlier this week on the subject.  Thinks that make you hmmmm.

How would your team react?

Any war stories or insights into good or bad reactions?

The following is based on a true story

Imagine you are hired to work as a software tester (or test lead) for a company that develops a web-based product .

When you get to the place you notice that these guys are really sharp and work with the latest technology and methodologies available; but very quickly you also notice a very strange behavior in the company: they run only a small number of tests before rolling new features out into production!

So you think to yourself: “EUREKA! I found the first thing to change in the company: release better products by running more tests before rolling stuff out!”

But when you take a closer look at the process and its numbers you realize two things:

1.  Even though some bugs are been released into production, their numbers are not a lot higher than in other places you’ve worked.  And what’s even more interesting is that whenever these bugs are released they are handled very quickly (in a matter of minutes) and with next-to-none effect on the users.

and

2.  If you were to introduce more testing into the system (even automation!) it would dramatically increase the cost and extend the development cycle delaying time to market to levels far from what the company is willing to accept.

So what can you do?

Well… apparently you should not go out and automatically dictate that more tests should be run!  Maybe you should start by understanding what they are doing and why?!

Sometimes exhaustive testing is not the best solution

It is pretty cool when you are working in ways that may seem unconventional and all-of-a-sudden someone shows you that other people are doing just about the same and writing about it in their blogs

I wanted to thank wonitta who last week pointed me to a post by Gojko Adzic where he wrote about companies that manage to reduce their exhaustive regression testing by using alternative approaches.  I won’t repeat what Gojko wrote, but I definitely recommend you read the article.

What I can do is explain how the company that I wrote about above successfully (IMHO) handles its development and testing process. They work with a number of principles that help them achieve their goal of very-fast-time-to-market while maintaining high levels of product quality.

No magic spells, only common sense and discipline

I don’t think these guys are singular, nor do they use any technique or technology that is out of reach to most development teams today.  They simply have evolved their process while trying to adapt to their highly competitive business environment, while looking for non-trivial ways of achieving their goals.

The process is not hard, but it does require discipline and the cooperation of all the team (developers, testers, product, etc).

Here are the principles I was able to identify from their process:

1. Short iterations & small incremental changes - by working on small agile cycles and making sure they break down large features into smaller iterations they are able to lower the complexity and the underlying risk of each release.

2. Good designs & risk analysis from the start - it is incredible how people always say that it is better to catch bugs in the design phase, but we still never do anything about it!  Right
Well, when you don’t have time to make mistakes it is better you work right from the start, and the best way to do this is by reviewing your assumptions and the design of your feature before you write your code.
By doing this these guys are able to make a good product (features, GUI, UX, integrations, etc) from the start; and by identifying and analyzing the high risk areas up-front they are able to design and write the feature in ways that lower these risks significantly.
The math is really simple:  LOWER RISK = LESS BUGS.

3. All the team tests - one thing about good developers is that they are not snobs!
In order to achieve high quality products this company has a whole-team-tests policy and this means that even though it is assumed that testers can perform tests better than programmers this doesn’t mean that the later should not run their own tests.
Testers help programmers define their tests and in the most risky features they also perform a large number of the test themselves, but there are also features where testers will not run a single test and all these tasks will fall upon the shoulders of the programmers themselves.

I think that this policy serves 2 main purposes, (1) it relieves  testers from being the bottlenecks in the process, and (2) it gives developers a higher sense of responsibility towards their programming standards (you take more careful steps on the tight-rope when you are your own safety net!).

4. Continuous integration - if you don’t know what this means then go and read more about it here.
The best thing about CI is that it allows the team to work on a continuously stable product, to find trivial regression bugs fast and to fix them even faster.

5. Organized pushes to production - releases and changes to production need to be controlled, performed gradually, and scheduled in advanced to make sure there are no conflicting pushes been done simultaneously.
Multiple changes to the same components, specially when done by different internal teams, automatically increase the risk of having bugs in production.

6. Gradual deployments – that allow you to deploy your code while keeping under control the effect they may have in your system.
By using techniques like A/B testing with limited percentages, or allowing only a subset of your users to access a specific change you are able to contain the effect of the changes to a small number of your users and thus limit the risk to your business.
These “production tests” are similar to Beta programs and provide the same type of information but a lot quicker!

7. Extensive monitoring - because when you have a bug in production you want to be the first to know about it and to get all the information up-front.
There are a number of different monitoring tools available, but putting them in place is only half the work.  You also need to develop your product in a way that it will integrate with the monitoring tools you use, and provide them with the information that will allow the team to understand the issue and fix for it right away.
In a sense is like having good-old Dr. Watson, but been able to solve the issues reported by your users RIGHT AWAY (instead of in the next version of Windows).

8. Predefined rollback and patching process – when you want to do something quickly and correctly under pressure you better define in advance what you want to do.  There is a saying in Hebrew that goes: “Work hard in practice, so that it comes easily during battle”.
The team should have a step-by-step rollback procedure with the scripts to run, the files to modify and all the rest of the operations required in order to quickly return the system to its last known good state (how it was before the release).  Since this operations may change from release to release it is necessary to create this procedure for each push (or at least to review the generic process and make the needed changes).
Remember also that rollbacks should not be the only option,  there should also be a defined procedure that defines what bugs can be “fixed in production” and how this should be done and tested.

9. Continuous learning and retrospective culture – because mistakes will be made, but you should learn from them in order not to commit them again.
The team has in place a process where each problem detected in production (bug, configuration issue, etc) is reviewed in a (semi) formal retrospective session, with lessons learned and action items to ensure the same issue won’t happen again.

WARNING:
Not all companies are created equally!

An important fact to understand is that no two companies are the same; and this is sometime true also for two groups or products within the same company.  Technology, company culture, and most importantly your users and the way they interact with your product, will define the way you develop and deploy your apps and the degrees of freedom you have as part of this process.

There are some industries where the price to pay for a mistake is really high, for example if you work on a life-sciences project where any bug can mean life-or-dead for a patient, or on the banking industry where a mistake can mean thousands or millions of dollars been mis-handled, or in aviation where bugs can mean a plane going down, then you should do exhaustive testing.

Still, the vast majority of companies working with web-based products don’t fall within the group above, and they have an intrinsic advantage that gives them a higher degree of freedom not available to firms that sell software that is installed in-house.

It is very easy to come up with many reasons why you cannot implement the process these guys use in your company, and you are probably right on most of them.  But you also need to take into account the business reality you work in, and realize that 95% of the companies world-wide can allow to have a bug or two in production once in a while, and balance this with the advantages of been able to release higher quality products and with shorter release cycles.

A final thought:

Maybe this is what we should aim for when we define our jobs as QA (Quality Assurance) Engineers and not as Testers?

After all there is no such thing as perfect & bug-free software, and most of us really want to make the best we can for the companies we work for…

Life taught me not to keep quiet and always say what I think is right out loud.  Life also taught me to surround myself with intelligent people who will not always agree with me, and who’s (constructive) criticism and (positive) arguments will challenge me to keep growing.  Most importantly life taught me that it’s OK to make mistakes, and it is even better to acknowledge and correct them as soon as I realize about them.

I believe it was Roosevelt who said: “The only man who makes no mistakes is the man who never does anything.”

Jerry & Lalit, many thanks !!

———————–

Some time ago I wrote a blog post called “Principles of good bug reporting“.  It talked about the basic things that make a good bug report and helps testers not to make the typical mistakes that will be criticized by other members in their team.   One of the points I mentioned is to always make sure you are giving bugs the correct Severity & Priority grades, since it is typical of inexperienced testers to give their bugs higher severity with the (sometimes unconscious) aim of having them corrected by the development team.

Jerry Weinberg commented about this blog saying among other things  “I don’t think testers should be prioritizing bugs“; and when I look back at what I wrote I agree with him, I was only half right on asking tester to set their severity and priority grades correctly.

A tester should know his business

The half that I was right was that a tester should be able to set the Severity of his bug correctly.

Severity should be (as much as possible) an objective measure of the damage caused by the specific bug to the Application Under Test (AUT).  And it is something a tester should be able to determine correctly.

The best way for a testers to set the severity of a bug correctly is by understanding enough of his product.
- What does the product do (and not only what is written in our site or in the marketing brochures)?
- Who uses the product and under what circumstances?
- What are the different flows a user can work with the product?
- How knowledgeable are the users in order to find workarounds to specific bugs?
And all the rest of the non-trivial information about the product and its users that will allow him to determine correctly how severe is a specific bug in the system.

I expect every tester, within a logical amount of time and with the correct guidance and information from his team, to reach this level of knowledge.  And the reason she needs to know the product this way is not only in order to set the severity right, this is the only way by which this person will be able to define, design and execute his tests correctly.

A tester should also know NOT to do the work of others in his team

On the other hand, and here is where the criticism from Jerry came, we should not require of a tester to determine the priority of a bug, remembering that priority is related to the time when a bug should be fixed, and it is only partially determined by its severity.

Let’s understand first of all that priority unlike severity is a subjective measure, subjective because it will be determined based on many personal (or team-related) factors and so it cannot be giving outside the context of the whole team.

What factors may affect the priority of a bug:

- Obviously one of the most important factors is the severity.  We will want to fix critical bugs before we solve the less critical bugs.

But there are many other factors such as:

- Complexity of the fix, we will want to solve complex fixes before we solve the trivial ones.  We do this in order to assure that a complex fix will be tested more and more thoroughly.  On the other hand sometimes we will choose not to solve a bug because there is not enough time to test it enough and so we prefer to release the bug as is than to solve it and introduce more severe bugs or delay the release of the version.

- Available resources.  Sometimes we choose to solve bugs based on the developers who are available, for example solve a simple GUI bug now and leave the complex DB bug for later due to the fact that the GUI developer is leaving on vacation next week, or because we need to send the screens now to the customer for validation.

- Bug fix clustering.  It is logical that if we are going to be working on a specific area of the code we try and fix as many bugs in it as possible, and so sometimes we will give bugs priority based on where we are already fixing other bugs.

- Earlier promises.  This is one of the most problematic, but also very easy to understand…  We will want to fix first the bugs we already promised to customers, since they are already aware of them and they are expecting them to be fixed.  So if we have 2 bugs with the same severity or even a bug that has a lower severity but was already promised to be fixed to a paying customer, we will most probably be taking care of it before we handle other bugs.

As you see, there are many reasons (all of them good and sensible) why we cannot set the priority of a bug, and why we should leave this to either the person who has all the information or at least make sure the decision is taken by the whole team in order to reflect as many of these points (and also the ones I left out) into consideration.

“Do you have tests that you still run manually, even though
they are also running as part of your automation test sets?

I am talking about the fact that even though you invested the time to automate something you still choose to run it manually.

Notice that I am asking for any tests, including the ones you run manually only once in a while or those you give to your junior testers to be 100% sure all is OK; in fact any tests you are still running both automatically and manually at the same time.

Surprisingly enough, even organizations with a relatively mature automation processes still run a significant number of their automatic scenarios as part of their manual tests on a regular basis, even when this doesn’t make any sense (at least on the theoretical level).

After realizing this was the case I sat down with a number of QA Managers (many of them PractiTest users) and asked them about the reason for this seemingly illogical behavior.

They provided a number of interesting reasons and I will go over some of them now:

We only run manually the tests that are really important

The answer that I got the most was that some teams choose tests to be run automatic and manually only when they are “really important or critical”.

This may sound logical at first, but on the other hand when you ask what is their criteria for selecting the tests that should be automated most companies say they select cases based on the number of times they will need to run them and also based on the criticality or importance of the business scenario.  In plain English, they automated the important test cases.

So if you choose to automate the test cases that are important then why do you still run them manually under the same excuse of them been really important…?  Am I the only one confused in here?

We don’t trust our automation 100%

The answer to the question I asked above, of why run the important tests even though they are already automated comes in the form of an even more interesting (and simple) answer: “We don’t really trust our test automation”

So this basically means they are investing 10 or even 50 man-moths of work and in most cases thousands of dollars on software and hardware in order to automate something and then they don’t really trust the results?  Where is the logic in this?

OK, so I’ve worked enough with tools such as QTP and Selenium in order to know that it is not trivial to write good and robust automation, but on the other hand if you are going to invest in automation you might as well do it seriously and write scripts that you can trust.  In the end it is a matter of deciding to invest on the platform and be serious in the work you are doing in order to get results you can trust (and I don’t mean buy expensive tools, selenium will work fine if you have a good infrastructure and write your scripts professionally).

The alternative is really simple, if you have automatic tests you can’t trust because they constantly give you wrong results (either false negative or even worst false positives!) you will eventually stop using them and finally throw all the work and money out the window…

We don’t know what is covered and what is not covered by the automated tests

This is also another big reason for why people waste time running manual tests that are already automated, they are simply not aware of which scenarios are included on their automation suite and which aren’t. In this situation they decide, based on their best judgement, to assume that “nothing is automated” and so run their manual test cases as if there was no automation.

If this is the case, then why do these companies have automation teams in the first place?

The automated tests are the responsibility of another team

Now for the interesting question, how come a test team “doesn’t know” which scenarios are automated and which aren’t?  The most common answer is that the tests are been written by a completely different team, a team of automation engineers, that is completely separate from the one running the manual tests.

Having 2 test teams, one manual and one automatic, is not something bad and in many cases it will be the best approach to achieve effective and trustworthy automation.  The bad thing is that these teams can sometimes be completely disconnected and so work on the same project without communicating and cooperating as it should be.

I will talk about how to communicate and cooperate in a future post, but the point here is that when you have 2 teams (one automated and one manual) you need to make an extra effort to make sure both teams are coordinated and as a minimum each of them know what the other is doing in order to plan accordingly.

We want to have all the results in a single place to give good reports

Finally, I wanted to mention a reason that was brought up by a number of test managers, even though they brought it as a difficulty and not a show stopper but it was brought up many times and so it sounded interesting enough to mention it.  The fact that they needed to provide a unified testing report for their project, and for this they either run part of their tests manually, or created manual tests to reflect the results of their automation.

Again, this looks like a simple and “relatively cheap” way of coordinating the process and even producing a unified report, but it suffers from the problem of being a repetitive manual job that needs to be done even after you already have an automation infrastructure and it will eventually but surely (specially as more and more automation is added) run into issues of coordination and maintenance that will make more expensive and in some cases will render it misleading or even obsolete.

What’s your take?

I am actively looking for more issues, experiences or comments like the ones above that revolve around the challenges in manual and automated testing.  Do you have stuff you want to share?  Please add it as comments or mail me directly to joel-at-practitest-com.

We’ve been working on a solution for these types of issues and so we are looking for all the inputs we can get in order to make sure it will provide an answer to as many of the existing challenges as possible.  I will be grateful for any help you can provide!

One of the articles I read includes a phrase that got stuck in my head, translating it freely to English it reads something like “If we look closely into the Agile Principles we can see that they are improvements made to some of the existing development methods with the use of experience and common sense…”

At the beginning I was not sure if I can agree with the statement, and then I decided to narrow my focus and look only into the testing aspects of the Agile organizations I know; can Agile testing be (basically) improvements made to the regular testing methodologies by applying the use of experience and common sense?

Is there such a thing as Agile Testing?

I’m not sure… is there?
Lately I think there really isn’t.

Is there really something you do as an Agile Tester that you couldn’t do as a “Regular or Traditional” Tester?  The answer, at least for me, is a definite NO.

Why wouldn’t I, as any kind of tester, be involved in the planning and scheduling stages for the project that is going to be developed by my programming and testing team?  Do I need to call it Planning Poker in order to take part of the process? NO.

Why can’t I as a tester on a V-Model team (if there is such a thing) work with the developers and help them test their features before they  are delivered to me for functional or integration testing?  Do I need to call it pair-programming or Agile Testing in order to make this collaboration happen? NO.

Should you as a tester (in any type of development team!) not work towards having in place the automation infrastructure that will allow your developers to test their changes constantly (CI) or institute a working methodology where Quality is the Responsibility of the Whole Team and not only of the testers at the end of process?  OF COURSE YOU SHOULD!

So in short there is nothing directly related to testing that should apply only to Agile Teams.

Agile Thinking instead of Agile Testing

Still, it would be naive to say that testers in Agile Teams don’t work differently than testers in non-agile teams. But I am starting to realize that this is mostly due to a change in mindset and not due to a change in the job description.

Agile teams simply think differently than non-agile teams.

For a start, my experience is that agile teams are more testing-conscious.  This means that programmers understand the real value of testing and so are more willing to do it than their non-agile colleagues.  Granted, most agile programmers will only go as far as extending their unit-testing coverage or maybe creating some additional functional tests using Cucumber or Selenium, but even this is great step forward.

Agile teams are also less hierarchical, and so developers are able to see testers as their peers, as colleagues who can give them feedback and with whom they can consult on professional issues (and not only as great palls to go out to lunch or for a beer once in a while, which is also good BTW!).

This last point is not trivial, the (hierarchical) differentiation between programmers and testers in traditional development teams is one of the biggest obstacles for a tester who wants to take a more active role in the development process.  This means that a tester in a traditional team will have a harder time been heard, but it doesn’t mean she shouldn’t raise her voice in order to voice her thoughts!  It’s a matter of initially trying harder to get noticed and making sure that whenever you do this the feedback you provide is intelligent and productive.

Lastly, a tester in a traditional team should not “be afraid” to take a more central role in the design of his application and the planning of his project.  Not only are we a part of the team who will be doing the job, but as testers we hold the central role of been the customer advocates within our teams.  The problem is that usually, due to the nature of traditional projects the most intensive part of the testing for project X happens almost in parallel to the most intensive part of the planning for project X+1.  The solution to this is only a matter of making it a priority to be part of the design and planning processes, and committing the time when it is useful.

So in the end, the way we want to work is basically up to us, you don’t need to be on an agile team to use an Agile Thinking mindset.

Well, sometimes you don’t have a choice, or you do but that choice is to release the application untested…

As part of a consulting engagement I am doing at a friend’s company, yesterday I started providing short training sessions for programmers who need to learn how to test better.  It’s not that this company doesn’t have testers, they have good testers!  But as many other agile teams they have much more testing tasks than available testers, and they want programmers to take part in at least some of their testing tasks.

Programmers are not good at testing!

A couple of months ago I wrote a post explaining why I think programmers make poor testers, and I still think that in a sense it is like asking a dog to fly (take a look at the cool picture in the post ). On the other hand I also believe that with some good intentions, hard work, and perseverance you can teach a dog to skip far enough and that way to jump over large obstacles.

In the same way, you will not be able to magically transform programmers into a good testers overnight, but you can start by working with them on their weaknesses and then teaching them some simple and effective testing techniques.  With good intentions, a little practice, some hand-holding, and constant feedback you will be able to get some good testing help in your project.

Step 1 – understand the limitation and weaknesses of a programmer

I started my session yesterday by going over the points I listed in my earlier blog that make programmers “less-than-perfect-testers”:

- Parental feeling towards their own code.
- Placing the focus mainly on positive scenarios, instead of actively looking for the bugs.
- Tendency to look at a “complex problem” as a collection of “smaller, simpler, and isolated cases”.
- Less End-to-End or User-Perspective oriented.
- Less experience and knowledge of the common bugs and application pitfalls.

It made a big difference during the session to not only list the weaknesses but talk about why each of them is naturally present in programmers due to the nature of their work, their knowledge and training.

Step 2 – how to plan tests

I’ve seen that many (most?) programmers have the perception that testing requires little or no planning.

Maybe we testers are guilty of this misconception since we don’t get them involved in our test planning process as much as we should, but the truth of the matter is that when we ask them to test they they automatically grab a mouse and start pressing on the buttons without much consideration or thought.

Test Planning is a cardinal aspect of good tested, so I gave them a couple of ideas and principles planing:

1.  DON’T TEST YOUR OWN CODE!
When as a team they are asked to test, they should make sure to divide the tasks so that each of them is testing the code developed by other programmers as much as possible.

2.  Work with your testing team to create test sets.
This specific team is using PractiTest (the hosted QA and Test Management platform my company is developing), but it can be any other tool or format (even word and excel!).  They should sit with their testers and define what test cases need to be run, reusing the testing scripts already available in their repositories.

3.  Expand your scenarios by making “Testing Lists”

- Features that were created or modified (directly or indirectly)
- User profiles and scenarios to be verified.
- Different environments / configurations / datasets to be tested

The use of these lists is two-fold.
They help you get a better idea of what you want to test while you are running your manual test scripts, and they also serve as a verification list to consult towards the end of the tests when you are looking for additional ideas ow when you want to make sure you are not missing anything of importance.

4.  Testing Heuristics – SFDEPOT (read San Francisco Depot)
The use of (good) heuristics greatly improve the quality of your testing.
I provided the programmers with the heuristic I learned fist and still helps me up to this day.  I read about SFDEPO(T) from James Bach some years ago – you can check one of the sources for it from Jame’s Site.

SFDEPOT stands for:
Structure (what the product is)
Function (what the produce does)
Data (what it processes)
Platform (what it depends upon)
Operations (how will it be used)
Time (when will it be used)

There are other heuristics and Mnemonics you can take from the Internet…

Step 3 – what to do when running tests

We talked a lot about tips to help perform good testing sessions.

1. Have a notebook handy.
In it you can take notes such as testing ideas you will want to test later, bugs you ran into and want to report later in order to “not to cut your testing thoughts”, etc.

2.  Work with extreme data.
Big files vs. Small files
Equivalent input classes [ -10 ; 0 ; 1 ; 10,000,000 ; 0.5 ; not-a-number ]
Dates: Yesterday, now, 10 years from now
etc

3.  Think about negative scenarios.
- How would Mr. Bean use your software?
- How would a hacker try to exploit the system?
- What would happen if…? (blackout, run out of space, exceptions, etc)
- What if your  2 year old would hijack the keyboard in the middle of an operation?
- etc

4.  Focus & Defocus.
(Thanks Ayal for teaching me this principle this year!)
I used this technique to explain to them that during their testing process they need to make a conscious effort to always look at the bigger picture (application, system, process) and not only focus on the specific function they were testing.

5.  Fight Inattentional Blindness.
I used the following film of the kids passing the balls to explain the concept of Inattentional Blindness and it worked great!
We were 8 people in the room, and only I had seen the video before.  Out of the 7 participants one is currently a tester, another one is a former tester turned programmer, the rest are “regular programmers”.
The cool thing is that only the 2 testers saw the Gorilla the first time…  talk about making a point!

Step 4 – what to do when (you think) you are done testing

We talked about how even when you think you are done testing you should always make sure there is nothing else that should be tested, and what techniques they can use in order to find these additional places:

1.  Walking the dog
Taking a break from your tests, doing another tasks for 30 – 60 minutes, and then returning to review the tests you did and what you found.  This break usually helps to refresh the mind and to come up with more ideas.

2.  Doing a walk-through session to review the tests you did.
Most of the time you will be able to get more ideas as you explain to your peers about tests you just did.
The funny part is that many of these ideas will come from yourself and the things you think about when you are explaining your tests to others out loud.

3.  Ask for ideas from other developers or testers.
Simple as it sounds, come to others in your team and ask them to pitch you ideas of stuff they think you could test.  90% of the stuff you will already have tested, but the other 10% might prove useful too!

In the end is a questions of mindset and motivation

One of the things I like most of agile teams (and many non-agile but yes smart development teams) is that they define Quality to be the responsibility of the whole team and not only of the testing guys running the tests tasks at the end of the process.

My final piece of advice to the group was that everything starts from them, and their understanding that testing is not a trivial task and definitely not a reprimand for doing a bad job or for finishing their tasks ahead of time.  What’s more, I am sure that once these guys start testing better and gaining a testing perspective of their application they will also start developing better software too.

Now I have another 3 development teams I need to train.
Let’s see how it goes with them

Have you ever been on a situation where you need to test 10 things but you have only the time for 6 of them, and on the other hand you look at your development team and 2 programmers are done with all their important tasks and started working on stuff that sits under the “nice-to-have” list (or even worst, they went to the game room to “test the Wii“)?  What do you do?

The answer will depend on how you have prepared your team and your testing infrastructure to delegate part of the testing task, in other words on how have you EMPOWERED your developers to be testers when needed.

But developers can’t test!!!

I know programmers are not ideal testers, I even wrote about it in the past. But when the alternative is that the features will go out untested or that to run a sanity test on all the features you will risk not running critical tests on more important parts of the system you may be willing to compromise.

The important thing then becomes how well have you built your testing infrastructure and how have you prepared your programmers so that they can lend a helping hand when needed.

It’s like teaching a dog how to fly

You cannot teach a dog how to fly, but you can train him to jump far enough. In the same way you cannot train a developer to become a star tester, but you can give him the ability to go over the system and find the “critical bugs”.

How do you do it? It’s not rocket science, although if it was it might actually be easier for them to understand it! Start by describing what you do when you approach a testing task.

I usually do 2 main things with my developers when I prepare them to test:

1. Explain about testing principles. How to look at the application from a user’s perspective and disregard as much as possible the internal architecture. I explain to them about building end-to-end workflows, about working with user profiles, thinking in terms of negative scenarios, and using extreme and wrong inputs in order to find for bugs.
I find this can be easily done by gathering them in a conference room and doing a quick presentation that will show them the different techniques and examples of the type of bugs they can expect to find with them.

2. Run Pair-Testing sessions. When I want to help a programmer understand how I think when I test I invite him to do a 30 to 90 minutes pair testing session. During this time I grab the mouse and explain to him how I think and what am I looking for as I test a part of the system. About half way through the session I switch places with him and ask him to “test out loud” so that I can help him and give him ideas and guidance.

These 2 activities help my programmers to embark on testing tasks with enough self confidence in order to take ownership on testing tasks.

Preparing the grounds

Another facilitating factor to empower your programmers to test is how ready your infrastructure is to support and guide them as they test. Is you test plan and your test-ware ready to be used by other people?

It is easier to go and tell a programmer to go and test Feature-X if you also provide him with some sanity tests and/or checklists that will guide him through the main parts of the process.   For this you need to write these tests or checklists ahead of time.

Personally I don’t think you need to create extremely documented steps, but it will certainly help if they have the main business scenarios available and they can go over the basic coverage of your application in a structured way.    Just make sure to explain to them that these scenarios are only a guiding path and the most important bugs will usually be found on the areas that are “on the sides” of the way and not necessarily along it (this is something that is easier explained as part of the pair testing walkthrough).

Another important aspect that is not always trivial is whether you have enough testing environments to accommodate all your new testers? Do you have enough licenses to test your external integrations?   Are your systems configured, up and running?

If you are not prepared to “host new testers” sometimes it may take even more time to set up these environments than what you can save by getting the extra help, so better be prepared ahead of time.

Lastly, how will your “new testers” document and report their tests so that you can review them after they are done? It may sound trivial but you want to make sure you can go over what they tested, review their work and provide feedback on additional stuff they may want to test “based on your professional experience”.

If you have a test management system (like PractiTest ) it may help, but if you don’t you can use text documents or even hand-writen notes. The important thing is that you can go over what they’ve done, understand what was achieved, and provide feedback if necessary.

In short, make sure your infrastructure is in place to incorporate more testers on-the-fly.

Understanding what to delegate and to who?

An additional point to review on this subject is related to understanding what parts of your testing tasks you can delegate and which ones are simply out of you “new testers” reach? What is complex and what isn’t? Where are the highest risk? etc.  As much as you want to relay on your programmers they simply don’t have the experience and in some cases the mind-set to test complex stuff and find all the issues in there, so better make your risk assessment correctly.

Also, you need to check who is going to be testing what feature or area of the system, for example you cannot give a feature to the same developer who wrote it (and you can check some of the reasons why on this article I mentioned before).

Learning to work with compromises

In the end it is a game of trade-offs, preparations and compromises.

It’s always nice to know you can call on additional resources when you need them, but understand they will be able to help you only in limited ways. Also understand that in order for this to work you need to prepare the infrastructure ahead of time and have “trained testers” who will be able to step to plate and help you out when you need them.

Today I took “half a day off” to work on developing a feature I’ve been wanting to add to PractiTest for a long time.

It’s fun to do it once in a while; to take a break, get my hands dirty, develop something for a bit and then go back to my regular work as a test manager and solution architect.  It’s not that I don’t trust my development team, I guess I did it for the fun of it

I read somewhere that it is related to changing your focus and “going deep” on a task that is unrelated to what you do all day but still closely linked to your work and how this helps to refresh your mind by breaking out of your working routine.

Writing GOOD CODE is hard!

For me as a tester it is also a reminder of how hard development work really is, a chance to experience first-hand how non-trivial it is to write something that looks good on all browsers and works correctly in all Operating Systems.

I am sure that even though I tested my “small feature” thoroughly, once it reaches someone from my team for testing he or she will find at least one or two annoying bugs on scenarios I had not thought about.  After all, isn’t this the art of testing?

Running GOOD TESTS is also hard!

The same is true about testing, you need to give a developer an end-to-end testing task for him to understand how difficult it is to test.

I heard this from a testing colleague in another company the other day, he works on an agile team where developers sometimes get testing tasks assigned to them, but then come back to him asking “if he can re-run the tests on his spare time” since they don’t feel god enough with their testing coverage.

Is this a case of someone wanting some reassurance and validation on his work, or someone who doesn’t want to take responsibility for his task and wants someone else to do it for him?  The truth is that it is hard to tell…

Teach your developers to test!

It is important for developers to take responsibility for end-to-end testing tasks once in a while; it helps them to understand what matters to us when we do our job, and maybe even more importantly it teaches them to do their jobs better!

If a programmer experiences first hand how his code behaves on a “realistic” testing environment then he can take this experience and code accordingly next time he sits to work on developing a feature.  It’s not about making him take responsibility for testing his own code, I think he simply can’t do this, but about learning to develop while been aware about additional factors that may affect his code.

To achieve this you cannot give him a 2 hours of testing task at the end of the sprint, he needs to take responsibility for a complete testing project and let him feel and live the work of a tester, even for a little while.

Doing this will not only provide you with some additional testing resources that can be used in times when the bottle neck sits in the QA department and Developers can help out, but it will also give them tools to stop bugs from crawling into the code in the first place.

Overall a win-win for the whole team.