Two weeks ago I published a blog called Manual and automated tests together are challenging that got a large number of replies and comments both here in the blog as well as in some LinkedIn groups.
To summarize the post, it talked about the fact that some QA teams that have automation in place still run part of their automatic test cases manually.
WHY do they do this? Some of the reasons I got were that (1) they like running some of their tests manually as well since they are “really important”, (2) that automation cannot be trusted 100%, (3) there is no clear definition or knowledge of what is exactly automated, and (4) they run test manually in order to update their test management system with the results (of their automation) and be able to generate complete and comprehensive reports for their management.
Some more reasons why to manually run your automatic test cases
So as I said, I got additional feedback from a number of people who read the post (thanks to everyone who got back to me!). Some of the most interesting points raised were the following:
1. Even though the test may look similar at first sight, they are testing different aspects of the same feature. For example an automatic test may verify the installation procedure on all O/S, and a “related manual test” will verify the effects of abnormal network traffic during the installation process.
2. There are some tests that are “semi-automatic” and not fully automatic, like blink-tests where you perform a number of operations and constantly take screen-captures of the application that are then reviewed quickly by the tester in order to detect GUI issues or bugs.
3. You can start additional manual tests on the places where your automation concluded. For example you can use automation to check the installation of the system and to validate the procedure to add, remove and modify data. Once these tests are done you can take the system created and continue running additional manual tests based on existing data and system setup.
Additional challenges when coordinating manual and automatic testing
Part of the feedback I got to the post was also in the way of additional challenges that raise in coordinating the work of your manual and automatic testing (not directly related to re-running your tests). Here are some of the most interesting points:
Figuring out what needs to be moved from manual to automatic testing
This is the first challenge and I guess one of the most significant decisions you need to make, since it will dictate the value you realize from your automation effort. I am not sure there is a text-book answer to this question, but I guess it needs to be related to 3 central factors:
ROI – the return on the investment to automate your test, or in plain English what will you gain from automating this specific script. This is usually a matter of how long it will take you to automate your script vs. how many times will you run it automatically and how much time do you save because of it.
Complexity of the Test – this refers to both how hard it is to create the automatic script but also how easy it is to do the test manually. There are tests that are simply too hard to do manually (e.g. testing the API of your product) and there are others that are practically impossible to automated (e.g. usability of a feature). You need to choose correctly what tests are more suited to be either automatic or manual.
Stability of your AUT – maybe the biggest challenge of any automatic test is to cope with changes on your AUT (Application Under Test). There are some automation platforms that do this better than others, but no automatic system will be able to “improvise and understand” in the way human-testers do. Since this is the case you need to add this factor to the list of things that will help you choose what to automate, and try not to work on areas of the product that you know will go through radical changes in the near future.
Coordinating the tasks between the manual and the automation testing teams
These are also big challenges that raise from the intrinsic differences between a good manual test engineer and a good automation engineer.
Who does what?
– Do the automation engineers define what to test or is this the call of the manual testers?
– Is scheduling tests a task of the manual testers or the automation testers?
– What happens when an automatic test fails, is the manual tester in charge of verifying if there is a bug or is this the job of the automation team?
All these questions and many more like them need to be defined up-front, and again here there is no text-book answer.
Personally I believe that a good automation engineer is closer to a developer than to a tester and so I like placing more weight on the tasks of my manual testers. Basically I ask my automation engineers to be “service providers” and to work with my manual testing engineers to supply them with the best possible automatic answer to their testing scenarios.
I also believe that automation is part of the complete testing effort and so the decision and responsibility of what to run and when to run it should be in the hands of the group in charge of the complete testing efforts (this is usually the task of the manual testers too).
Lifting the tabu from test automation
I really liked this comment from Marco Venzelaaer in LinkedIn: “Some automation teams make test automation a ‘dark art’ which is closely guarded within the team…”
This point got me laughing but also grabbing my head as he managed to articulate something I had felt for a number of years. Some automation engineers feel that by treating their work as something mysteriously hard and extremely complex they gain some sort of work security. They are specially successful by seeding these feelings on manual testers, who don’t have any coding experience on their side and so believe what their automation colleagues are telling them.
Other than a false feeling of job security, these automation engineers also manage to generate a sort of tabu around their work that will make manual testers refrain from trying to understand it or have any influence on the process. In the end this behavior is harmful to the coordination of efforts and has a deterrent effect on the overall achievements of the testing team.
So how do you coordinate the work of you automated and manual testing in our team?
I think that the key to maintaining a healthy manual-vs-automation relation in testing is TEAMWORK & COMMUNICATION
You need to make sure both teams, the manual testers and the automation engineers, are working based on the same agenda, coordinating their tasks based on the same priorities, and understanding how each player helps the other to fulfill the goals of the organization. In simple therms, how they work together in order to make their testing process faster, broader and eventually more effective.
Trivial things to take into account are:
1. Make sure they are coordinated, by having regular update meetings and making sure there is active participation of both teams when planning the testing tasks for each project.
2. Have both teams work on an integrated environment where both manual and automated tests are managed and executed. This will allow all your testers, and also every other person in your company, to see the “full picture” and understand both on a planning and execution level what is been covered, what has been tested, and what are the results of these tests. After all no one really cares if the test was run manually or automatically, they care about the results of the test.
3. Have a process in place where automation is a tool to hep your manual testing team.
The correct process is what will eventually make or break the relationship between manual and automatic tests. Both teams need understand that the idea of automation is to free the time of manual testers to run the more complex tests, those that are hard or expensive to automated. Once they understand that they compliment each other and not compete with one-another they will be able to focus on their shared challenges instead of their rivalries.
In the end, it is the work of the manager and the Organization to create the environment and the “team rules” that will allow all members to feel like they provide their work and help drive the project forward TOGETHER.
Practitest is an end-to-end test management tool, that gives you control of the entire testing process - from manual testing to automated testing and CI.
Designed for testers by testers, PractiTest can be customized to your team's ever-changing needs.
With fast professional and methodological support, you can make the most of your time and release products quickly and successfully to meet your user’s needs.