Two weeks ago I published a blog called Manual and automated tests together are challenging that got a large number of replies and comments both here in the blog as well as in some LinkedIn groups.

To summarize the post, it talked about the fact that some QA teams that have automation in place still run part of their automatic test cases manually.

WHY do they do this?  Some of the reasons I got were that (1) they like running some of their tests manually as well since they are “really important”, (2) that automation cannot be trusted 100%, (3) there is no clear definition or knowledge of what is exactly automated, and (4) they run test manually in order to update their test management system with the results (of their automation) and be able to generate complete and comprehensive reports for their management.

Some more reasons why to manually run your automatic test cases

So as I said, I got additional feedback from a number of people who read the post (thanks to everyone who got back to me!).  Some of the most interesting points raised were the following:

1. Even though the test may look similar at first sight, they are testing different aspects of the same feature.  For example an automatic test may verify the installation procedure on all O/S, and a “related manual test” will verify the effects of abnormal network traffic during the installation process.

2. There are some tests that are “semi-automatic” and not fully automatic, like blink-tests where you perform a number of operations and constantly take screen-captures of the application that are then reviewed quickly by the tester in order to detect GUI issues or bugs.

3. You can start additional manual tests on the places where your automation concluded.  For example you can use automation to check the installation of the system and to validate the procedure to add, remove and modify data.  Once these tests are done you can take the system created and continue running additional manual tests based on existing data and system setup.

Additional challenges when coordinating manual and automatic testing

Part of the feedback I got to the post was also in the way of additional challenges that raise in coordinating the work of your manual and automatic testing (not directly related to re-running your tests).  Here are some of the most interesting points:

Figuring out what needs to be moved from manual to automatic testing

This is the first challenge and I guess one of the most significant decisions you need to make, since it will dictate the value you realize from your automation effort.  I am not sure there is a text-book answer to this question, but I guess it needs to be related to 3 central factors:

ROI – the return on the investment to automate your test, or in plain English what will you gain from automating this specific script.  This is usually a matter of how long it will take you to automate your script vs. how many times will you run it automatically and how much time do you save because of it.

Complexity of the Test – this refers to both how hard it is to create the automatic script but also how easy it is to do the test manually.  There are tests that are simply too hard to do manually (e.g. testing the API of your product) and there are others that are practically impossible to automated (e.g. usability of a feature).   You need to choose correctly what tests are more suited to be either automatic or manual.

Stability of your AUT – maybe the biggest challenge of any automatic test is to cope with changes on your AUT (Application Under Test).  There are some automation platforms that do this better than others, but no automatic system will be able to “improvise and understand” in the way human-testers do.  Since this is the case you need to add this factor to the list of things that will help you choose what to automate, and try not to work on areas of the product that you know will go through radical changes in the near future.

Coordinating the tasks between the manual and the automation testing teams

These are also big challenges that raise from the intrinsic differences between a good manual test engineer and a good automation engineer.

Who does what?
- Do the automation engineers define what to test or is this the call of the manual testers?
- Is scheduling tests a task of the manual testers or the automation testers?
- What happens when an automatic test fails, is the manual tester in charge of verifying if there is a bug or is this the job of the automation team?

All these questions and many more like them need to be defined up-front, and again here there is no text-book answer.

Personally I believe that a good automation engineer is closer to a developer than to a tester and so I like placing more weight on the tasks of my manual testers.  Basically I ask my automation engineers to be “service providers” and to work with my manual testing engineers to supply them with the best possible automatic answer to their testing scenarios.

I also believe that automation is part of the complete testing effort and so the decision and responsibility of what to run and when to run it should be in the hands of the group in charge of the complete testing efforts (this is usually the task of the manual testers too).

Lifting the tabu from test automation

I really liked this comment from Marco Venzelaaer in LinkedIn: “Some automation teams make test automation a ‘dark art’ which is closely guarded within the team…”

This point got me laughing but also grabbing my head as he managed to articulate something I had felt for a number of years.  Some automation engineers feel that by treating their work as something mysteriously hard and extremely complex they gain some sort of work security. They are specially successful by seeding these feelings on manual testers, who don’t have any coding experience on their side and so believe what their automation colleagues are telling them.

Other than a false feeling of job security, these automation engineers also manage to generate a sort of tabu around their work that will make manual testers refrain from trying to understand it or have any influence on the process.  In the end this behavior is harmful to the coordination of efforts and has a deterrent effect on the overall achievements of the testing team.

So how do you coordinate the work of you automated and manual testing in our team?

I think that the key to maintaining a healthy manual-vs-automation relation in testing is TEAMWORK & COMMUNICATION

You need to make sure both teams, the manual testers and the automation engineers, are working based on the same agenda, coordinating their tasks based on the same priorities, and understanding how each player helps the other to fulfill the goals of the organization.  In simple therms, how they work together in order to make their testing process faster, broader and eventually more effective.

Trivial things to take into account are:

1.  Make sure they are coordinated, by having regular update meetings and making sure there is active participation of both teams when planning the testing tasks for each project.

2. Have both teams work on an integrated environment where both manual and automated tests are managed and executed.  This will allow all your testers, and also every other person in your company, to see the “full picture” and understand both on a planning and execution level what is been covered, what has been tested, and what are the results of these tests.  After all no one really cares if the test was run manually or automatically, they care about the results of the test.

3. Have a process in place where automation is a tool to hep your manual testing team.
The correct process is what will eventually make or break the relationship between manual and automatic tests.  Both teams need understand that the idea of automation is to free the time of manual testers to run the more complex tests, those that are hard or expensive to automated.  Once they understand that they compliment each other and not compete with one-another they will be able to focus on their shared challenges instead of their rivalries.

In short you need to make sure your teams have both the agenda as well as the infrastructure required in order to coordinate their work.

In the end, it is the work of the manager and the Organization to create the environment and the “team rules” that will allow all members to feel like they provide their work and help drive the project forward TOGETHER.

“Do you have tests that you still run manually, even though
they are also running as part of your automation test sets?

I am talking about the fact that even though you invested the time to automate something you still choose to run it manually.

Notice that I am asking for any tests, including the ones you run manually only once in a while or those you give to your junior testers to be 100% sure all is OK; in fact any tests you are still running both automatically and manually at the same time.

Surprisingly enough, even organizations with a relatively mature automation processes still run a significant number of their automatic scenarios as part of their manual tests on a regular basis, even when this doesn’t make any sense (at least on the theoretical level).

After realizing this was the case I sat down with a number of QA Managers (many of them PractiTest users) and asked them about the reason for this seemingly illogical behavior.

They provided a number of interesting reasons and I will go over some of them now:

We only run manually the tests that are really important

The answer that I got the most was that some teams choose tests to be run automatic and manually only when they are “really important or critical”.

This may sound logical at first, but on the other hand when you ask what is their criteria for selecting the tests that should be automated most companies say they select cases based on the number of times they will need to run them and also based on the criticality or importance of the business scenario.  In plain English, they automated the important test cases.

So if you choose to automate the test cases that are important then why do you still run them manually under the same excuse of them been really important…?  Am I the only one confused in here?

We don’t trust our automation 100%

The answer to the question I asked above, of why run the important tests even though they are already automated comes in the form of an even more interesting (and simple) answer: “We don’t really trust our test automation”

So this basically means they are investing 10 or even 50 man-moths of work and in most cases thousands of dollars on software and hardware in order to automate something and then they don’t really trust the results?  Where is the logic in this?

OK, so I’ve worked enough with tools such as QTP and Selenium in order to know that it is not trivial to write good and robust automation, but on the other hand if you are going to invest in automation you might as well do it seriously and write scripts that you can trust.  In the end it is a matter of deciding to invest on the platform and be serious in the work you are doing in order to get results you can trust (and I don’t mean buy expensive tools, selenium will work fine if you have a good infrastructure and write your scripts professionally).

The alternative is really simple, if you have automatic tests you can’t trust because they constantly give you wrong results (either false negative or even worst false positives!) you will eventually stop using them and finally throw all the work and money out the window…

We don’t know what is covered and what is not covered by the automated tests

This is also another big reason for why people waste time running manual tests that are already automated, they are simply not aware of which scenarios are included on their automation suite and which aren’t. In this situation they decide, based on their best judgement, to assume that “nothing is automated” and so run their manual test cases as if there was no automation.

If this is the case, then why do these companies have automation teams in the first place?

The automated tests are the responsibility of another team

Now for the interesting question, how come a test team “doesn’t know” which scenarios are automated and which aren’t?  The most common answer is that the tests are been written by a completely different team, a team of automation engineers, that is completely separate from the one running the manual tests.

Having 2 test teams, one manual and one automatic, is not something bad and in many cases it will be the best approach to achieve effective and trustworthy automation.  The bad thing is that these teams can sometimes be completely disconnected and so work on the same project without communicating and cooperating as it should be.

I will talk about how to communicate and cooperate in a future post, but the point here is that when you have 2 teams (one automated and one manual) you need to make an extra effort to make sure both teams are coordinated and as a minimum each of them know what the other is doing in order to plan accordingly.

We want to have all the results in a single place to give good reports

Finally, I wanted to mention a reason that was brought up by a number of test managers, even though they brought it as a difficulty and not a show stopper but it was brought up many times and so it sounded interesting enough to mention it.  The fact that they needed to provide a unified testing report for their project, and for this they either run part of their tests manually, or created manual tests to reflect the results of their automation.

Again, this looks like a simple and “relatively cheap” way of coordinating the process and even producing a unified report, but it suffers from the problem of being a repetitive manual job that needs to be done even after you already have an automation infrastructure and it will eventually but surely (specially as more and more automation is added) run into issues of coordination and maintenance that will make more expensive and in some cases will render it misleading or even obsolete.

What’s your take?

I am actively looking for more issues, experiences or comments like the ones above that revolve around the challenges in manual and automated testing.  Do you have stuff you want to share?  Please add it as comments or mail me directly to joel-at-practitest-com.

We’ve been working on a solution for these types of issues and so we are looking for all the inputs we can get in order to make sure it will provide an answer to as many of the existing challenges as possible.  I will be grateful for any help you can provide!

All of these are important questions, and even if there is no text-book answer for them there are some guidelines you can consider when wondering about this subject.

My thoughts on test sizes were prompted by 2 separate events. This week we are starting a long-range project to re-factor PractiTest’s test automation framework (the one we use to test ourselves!). This time we are making a conscious effort to keep our tests as short as possible and to re-use them as part of more complex test-sets.

I am also guiding one of our PractiTest users on his custom deployment, and we are trying to define how deep and extensive should the tests be in order to answer his needs and constraints. With this customer we are reaching the conclusion that most of the tests should be kept small and high level; but still a limited number of regression scenarios will be written very detailed and covering the system from end to end.

When considering the right size and depth of your test the spectrum is extremely wide.  On the one hand you can write a word-by-word scripted test, allowing you to take ‘any person from the street’ and use him to test an application he hasn’t ever seen.  On the other hand you can give an application to a tester and ask for her “professional opinion” without giving her any indication to what the AUT should do or what you want to check, then come back 3 days later to see what she has to say.

I personally think that both of these extremes are wrong! But there is a large space in the middle to choose from, and many times the right choice will not be easy to find or trivially clear.

I personally prefer my tests to be small and as high-level as possible. I believe that if I take good testers and we give them an introduction on what the AUT should do and who it should serve, then I don’t need to provide them with step by steps cases in order for them to do the job right.

Instead of creating extensive test cases I prefer to user generic check-lists and heuristics to help me go over my application. These tools allow me and my testers to test thoroughly and to find the important stuff; but on the other hand it doesn’t drive us mad by writing kilometric tests with low-level steps that no one really needs and just frustrate testers simply by needing to read them.

There is also an added point to having high-level tests, they force us to think when we test!  I’ve seen that if you give even an experienced tester a very detailed test she will turn into a kind of live-test-automation-tool, performing each and every step but being so focused on it that she doesn’t pay attention to the “interesting scenarios” lying right next to her predefined test.

When your tests are high-level steps or even charters or threats (like a couple of interesting articles I read this week about Exploratory Thread-based testing by the Bach Brothers – here are both by James & Jon) you actually motivate your testers to THINK BIG and take more responsibility for the test execution, overall achieving better results and finding more interesting and severe bugs.

On the other hand, having no tests at all and trusting on the intuition of your testers is a sure recipe for failure and important escaping defects; specially when not all your testers are experts in all the system, and even worst if they need to work under pressure when needing to provide results quickly.

So I guess that at least for me size does matter in my tests! I try to keep them small and smart as possible.

What do you think?

I started working on an automation project in a company where they recently adopted a strong and healthy agile development approach.  One of the first things they did was to distribute the once-unified testing team and incorporate the testers as part of the organic development teams; up to now all good and fine.

The VP R&D, who I know from a previous company we both worked together ages ago, brought me in to solve a problem that started when each of the teams began developing their test automation, all in their own separate ways, and each one started running into different issues halting their progress almost simultaneously.

I will be sincere and say that in my own egocentric way I thought the answer would be trivial and based on the models I had used in the past, where you create a central team in charge of automation and they provide services to the rest of the teams in the most professional and effective way.

Sounds logical, right?  At least it did to me.
So here is where I came (to save the day!?!?!).

The first thing I did was to set expectations.
I am only a Testing Guy with some experience, not a Magician, Miracle Worker, or a Messiah that will solve all the issues by imploring the help of any Testing Deity above (or even bellow).  The best I can do is try to use some of my experience and a lot of common sense (both mine and theirs) to find the way to work correctly.  I hope they understood this, if not the road is going to be really bumpy…

The second thing I did was to list the stakeholders within each team and across the company that I should talk to in order to get a good and balanced image of was is going on.

Here is where it started getting interesting, as things usually do when you start going deep into the bits and bites and asking some interesting questions.  BTW, for this I really recommend using an approach based on the 5 Y’s.

I quickly realized 2 main things, a bad one and a very good one, that were happening in the teams.

The bad thing(/s):
Since the automation had suddenly become a tasks of the Development Team Leaders, and since the Company did not have a single person who had ever been part of a Test Automation Process, they were running into all sorts of “rookie automation” issues.

• Wrong test organization
• Teams not sharing or even aware of the tests being developed by the other teams
• Lack of a proper test management or execution platform
• Lack of test development experience
• Tests that were not written in a modular way
• Tests that, when they fail, don’t give enough information about the nature of the error
• etc.

Just think about all the possible “simple” mistakes in the book, and most probably they were there in one team or another.

The (very) positive thing I saw was that the Team Leaders, who are all R&D managers in one way or another, really took ownership of the testing tasks and their automation.  Not only did they take ownership, but gave it a very high priority within the tasks of their teams, so much so that they were even giving automation development tasks to their brightest developers and team members.

Here is where I realized that I needed to let go of the traditional approach I had in mind.

Even though my initial feeling was to go by the traditional way and solve the issue by creating a centralized team to handle the test automation, I decided that this approach would very much eliminate the involvement of the Team Leaders who had really done the mental switch to adopt testing as part of their core responsibilities.

So, this week when I came back to the VP R&D with my analysis and recommendations I actually presented him a proposal that will take me out of my “known & comfort zone” but will allow us to leverage the great agile spirit currently present in the company.  Instead of creating a team that will develop automation, I will start working with each of the teams in order to solve their issues, and we will create an “informal” automation forum that will be in charge of correlating, sharing, and levering the automation process between the teams.

We will start working in the next 2 weeks so wish me luck!
I’ll keep posting updates and interesting stuff as they will surely become available.