Sometimes measuring only one thing
can be worst than not measuring anything at all.

A QA colleague forwarded me a report he sent to his management where he pointed at the fact that the whole R&D Organization was not delivering products at the pace they used to do it in the past.

He was airing out a “known” general concern shared by many testers and developers, and to do this he searched for some “hard data” that would prove this point.

The report was very simple, and it compared the number of releases by all the teams for the last period vs. the period immediately before this one; and it showed that in the last period all the teams together had only delivered about 40% of the releases they did previously.

Simple, right? – WRONG!

What happened with the report?

Immediately after he sent the report, the Team Leads of two of the development teams answered with short emails “explaining” that in the last period they had also had a number of holidays, and attributing everything to this fact.

With this excuse the value the report went down and the subject was erased from Management’s mind and agenda…

The “funny thing” is that my friend had also seen this point, but he thought everyone would see the difference was too large in order to be caused solely because of the holidays!

What can we learn?

There are some basic lessons we can learn from this issue that apply to most organizations

1. Don’t use only one number in order to draw a complete picture

A basic mistake made by many testers and QA Managers is to base all their conclusions on only one number or metric. Basing everything in one measurement is like reaching your conclusions based on only one side of the story instead of collecting information from all possible sides.

So what do you need to do? Look for more than one angle in order to draw a complete picture.

In the case above, the manager could have looked for information about the number of issues detected during the process, or amount of rejections from the field, or at least start by “normalizing” the number of deliveries based on the total number of working days in both periods.

2. Take into account your workplace politics

It is naive to thing that you will send a report pointing at a serious problem and whoever is responsible for it will simply step forward and happily take the responsibility. Victories have thousands of parents, while defeats are usually orphans.

This means that whenever you point to an issue in the system you need to make sure to cover all your angles and to provide information that is, as much as possible, irrefutable.

An important (& ethical) thing to do is to contact the person who may be responsible for the issue and make sure you communicate your findings to him ahead of time, before sending out the report to everyone else. This will give him a chance to come up with ideas on what caused this and even provide as some solutions that you can include in your report.

3. Metrics should include comparable historical information

In the case above, what could have taken down the argument about the holidays?
To compare the number of deliverables not to the period before, but to the same period last year!

Not all metrics are trivial by themselves, and so you may need to compare information in order to see the real differences (or similarities) between them.

In the case of time-based measurements, you need to think about seasonality factors such as:
- Season or time of year
- Part of the quarter
- Time of day
etc.

4. Take into account external factors

As much as you want to be able to measure everything with raw numbers there will always be things you cannot measure that affect your process. You need to research these factors and take them into account in your report.

In the case above, it would have been as simple as stating that in the last period there were also some out of the ordinary holidays, and then add a note saying that even if you factor them into account they would not lower the productivity bellow 80% of the regular level, leaving another 40% decrease in productivity unaccounted for.

External factors will vary all the time, they are the out-of-the-ordinary things that will be trivially visible for people who are living the process. Some examples can be:
- Extraordinary team growth or reduction
- Extreme product shifts or market disruptions
- Things like prolonged strikes
- And even in some cases, as I saw not too long ago in a company, unusual number of births in a team (5 out of 8 team members received a child in a period of 3 months)

Bottom line

One of the main difference between a credible story and plain gossip is that a story has more than angle to it. Your metric’s-based-report is basically a way to tell a story of what is happening in your product/process/team.

So, in your next report or fact-based email make sure you take into account:
1. To have more than one metric or angle measured
2. Company politics
3. Historical information
and
4. Extraordinary factors

Two weeks ago I published a blog called Manual and automated tests together are challenging that got a large number of replies and comments both here in the blog as well as in some LinkedIn groups.

To summarize the post, it talked about the fact that some QA teams that have automation in place still run part of their automatic test cases manually.

WHY do they do this?  Some of the reasons I got were that (1) they like running some of their tests manually as well since they are “really important”, (2) that automation cannot be trusted 100%, (3) there is no clear definition or knowledge of what is exactly automated, and (4) they run test manually in order to update their test management system with the results (of their automation) and be able to generate complete and comprehensive reports for their management.

Some more reasons why to manually run your automatic test cases

So as I said, I got additional feedback from a number of people who read the post (thanks to everyone who got back to me!).  Some of the most interesting points raised were the following:

1. Even though the test may look similar at first sight, they are testing different aspects of the same feature.  For example an automatic test may verify the installation procedure on all O/S, and a “related manual test” will verify the effects of abnormal network traffic during the installation process.

2. There are some tests that are “semi-automatic” and not fully automatic, like blink-tests where you perform a number of operations and constantly take screen-captures of the application that are then reviewed quickly by the tester in order to detect GUI issues or bugs.

3. You can start additional manual tests on the places where your automation concluded.  For example you can use automation to check the installation of the system and to validate the procedure to add, remove and modify data.  Once these tests are done you can take the system created and continue running additional manual tests based on existing data and system setup.

Additional challenges when coordinating manual and automatic testing

Part of the feedback I got to the post was also in the way of additional challenges that raise in coordinating the work of your manual and automatic testing (not directly related to re-running your tests).  Here are some of the most interesting points:

Figuring out what needs to be moved from manual to automatic testing

This is the first challenge and I guess one of the most significant decisions you need to make, since it will dictate the value you realize from your automation effort.  I am not sure there is a text-book answer to this question, but I guess it needs to be related to 3 central factors:

ROI – the return on the investment to automate your test, or in plain English what will you gain from automating this specific script.  This is usually a matter of how long it will take you to automate your script vs. how many times will you run it automatically and how much time do you save because of it.

Complexity of the Test – this refers to both how hard it is to create the automatic script but also how easy it is to do the test manually.  There are tests that are simply too hard to do manually (e.g. testing the API of your product) and there are others that are practically impossible to automated (e.g. usability of a feature).   You need to choose correctly what tests are more suited to be either automatic or manual.

Stability of your AUT – maybe the biggest challenge of any automatic test is to cope with changes on your AUT (Application Under Test).  There are some automation platforms that do this better than others, but no automatic system will be able to “improvise and understand” in the way human-testers do.  Since this is the case you need to add this factor to the list of things that will help you choose what to automate, and try not to work on areas of the product that you know will go through radical changes in the near future.

Coordinating the tasks between the manual and the automation testing teams

These are also big challenges that raise from the intrinsic differences between a good manual test engineer and a good automation engineer.

Who does what?
- Do the automation engineers define what to test or is this the call of the manual testers?
- Is scheduling tests a task of the manual testers or the automation testers?
- What happens when an automatic test fails, is the manual tester in charge of verifying if there is a bug or is this the job of the automation team?

All these questions and many more like them need to be defined up-front, and again here there is no text-book answer.

Personally I believe that a good automation engineer is closer to a developer than to a tester and so I like placing more weight on the tasks of my manual testers.  Basically I ask my automation engineers to be “service providers” and to work with my manual testing engineers to supply them with the best possible automatic answer to their testing scenarios.

I also believe that automation is part of the complete testing effort and so the decision and responsibility of what to run and when to run it should be in the hands of the group in charge of the complete testing efforts (this is usually the task of the manual testers too).

Lifting the tabu from test automation

I really liked this comment from Marco Venzelaaer in LinkedIn: “Some automation teams make test automation a ‘dark art’ which is closely guarded within the team…”

This point got me laughing but also grabbing my head as he managed to articulate something I had felt for a number of years.  Some automation engineers feel that by treating their work as something mysteriously hard and extremely complex they gain some sort of work security. They are specially successful by seeding these feelings on manual testers, who don’t have any coding experience on their side and so believe what their automation colleagues are telling them.

Other than a false feeling of job security, these automation engineers also manage to generate a sort of tabu around their work that will make manual testers refrain from trying to understand it or have any influence on the process.  In the end this behavior is harmful to the coordination of efforts and has a deterrent effect on the overall achievements of the testing team.

So how do you coordinate the work of you automated and manual testing in our team?

I think that the key to maintaining a healthy manual-vs-automation relation in testing is TEAMWORK & COMMUNICATION

You need to make sure both teams, the manual testers and the automation engineers, are working based on the same agenda, coordinating their tasks based on the same priorities, and understanding how each player helps the other to fulfill the goals of the organization.  In simple therms, how they work together in order to make their testing process faster, broader and eventually more effective.

Trivial things to take into account are:

1.  Make sure they are coordinated, by having regular update meetings and making sure there is active participation of both teams when planning the testing tasks for each project.

2. Have both teams work on an integrated environment where both manual and automated tests are managed and executed.  This will allow all your testers, and also every other person in your company, to see the “full picture” and understand both on a planning and execution level what is been covered, what has been tested, and what are the results of these tests.  After all no one really cares if the test was run manually or automatically, they care about the results of the test.

3. Have a process in place where automation is a tool to hep your manual testing team.
The correct process is what will eventually make or break the relationship between manual and automatic tests.  Both teams need understand that the idea of automation is to free the time of manual testers to run the more complex tests, those that are hard or expensive to automated.  Once they understand that they compliment each other and not compete with one-another they will be able to focus on their shared challenges instead of their rivalries.

In short you need to make sure your teams have both the agenda as well as the infrastructure required in order to coordinate their work.

In the end, it is the work of the manager and the Organization to create the environment and the “team rules” that will allow all members to feel like they provide their work and help drive the project forward TOGETHER.

There’s the challenge of working with new technologies, new languages and even new platforms; forcing us to investigate and learn how to cope with these new paradigms.  But in a sense all these technical issues can be easily solved by turning to the best-friend of most geeks (like me).

Chances are that if you are asking yourself what tools to use in order to automatically test a new technology, how to write tests for a specific scripting language, or what are the weak spots on an operating system or mobile platform; the answers will already be available on the Internet, written by some early adopter or tester who took on the challenge and posted a paper or blog explaining what he did and how.

But the real challenge is coping with the things that cannot be searched on the Internet, and the ones you will need to figure out by yourself. I am referring to the functional and business aspects of your new application and the way your users will work and interact with your product once it is released.

Who do I  talk to in order to get information?

The first question to ask is “Who do you need to talk to in order understand what to test?” and the answer, at least in the beginning of the process, is simple – TALK TO EVERYONE YOU CAN.

When you start a new project or when you’re asked to test a new application you need to start by mapping all the people who can provide you with information.  So make a list of everyone you can talk to – Marketing, Development, Sales, Support, the CEO – and try to talk to each of them in order to understand what you need to test (and why)!!!???

Even if you don’t manage to talk to all of them (sometimes it can be tricky to get a 30 min session with your CEO…) at least try to make the list and cover as many different people as possible in order to get different perspectives and testing inputs.

What questions to ask?

Here is the hard part of the problem…

Imagine the following scenario:

You finally schedule the meeting with your CEO in order to get her inputs into what should be tested in the system. You arrive at the meeting and ask her: “Mrs. CEO, I have been given the task of testing the new product, and I wanted to ask you what do you think should be tested?”.
She stops and looks at you for a couple of seconds before answering: “Well, I don’t know, aren’t you the testing expert here…?  I guess you better test everything, right?  We don’t want any bugs slipping out the door, do we?”

What’s wrong with the scenario above?  Well, basically we came to the person and asked him the wrong questions…

If you pay attention to what the CEO said she was right, we are the testing experts.  She expects you to come up with what should be tested in the system, but on the other hand she is also living under the illusion that everything should or even can be tested (something we all know is impossible and even when possible, not economical in the long run).

The problem here is that we asked someone else to do our work for us, to come up with what to test, instead of asking for the input we need to understand by ourselves what needs to be tested and how.

So going back to our CEO meeting scenario, what questions would have been valuable to ask?  Well, you need to ask for her inputs on the system, without even talking about the testing operations.  Try to understand what areas are important from a user perspective, or based on our competitive advantages, or based on what makes our application unique, etc.  You should focus your questions on what is important for them.

Here are a couple of examples you can ask your CEO or VP Marketing or even Director of Sales:
- What are the most important aspects of our Product?  The things that make us unique?
- What areas in the product are the most widely used by our customers?  What type of things would make our users angry and make them choose not to work with our product?
- Where is the market focusing on today?
- How are we better than our competitors?  What areas are the ones that are the most problematic in our competitors, the same areas where we want to exceed?
- Are there any risks you think we should be specially aware off?  Risks in our technology, risks in the product?

Notice that we didn’t ask them what to test, but we did ask what’s important in their eyes (based on their experience).

In the case of the CEO, Marketing and Sales functions we will want to talk about stuff that relates to the functionality of the product.

If we were to talk to our Support Team we would ask them questions related to the areas in which our users find bugs, focusing both on the places where there are the largest amount of bugs as well as where the most critical issues are found.

Finally, when talking to our development peers we will ask them about technological risks, as well as places where they are making the most changes, or where the product is relatively weak or complex and so where we should be putting more testing efforts.

The art of listening (and putting together the puzzle)

So what do you do with all the information?  Basically you need to take the stuff you got, and process it in order to get a 360-degree reading of your application.  What do I mean by 360-degrees?  I mean from all the different angles that matter: Technology, Usability, Supportability, Competitiveness, etc.

After all, your work is to test and to provide visibility into whether the application is meeting the quality standards of your users and stakeholders.  The only way to do this is by understanding what’s important to all of them and creating a test plan (or work plan!) that will effectively cover it.

We all love our programmers very much!  All of us are part of a superb development team and we are sure they do a great job.  And maybe most importantly, all of us (programmers and testers alike) only want the best for our users and the applications we develop.

Still, every once in a while, as testers we will run into situations where we realize our team is making a wrong decision and something should be done to correct it.  Whether this happens once a week or once a quarter varies, but I can promise you that wherever testers and programmers work together there will always be points of friction and subjects where we don’t see eye-to-eye.  The question is what do you do when stuff like this happens?

Saying “There’s nothing I can do!” is only the easy way out

Last week I was doing a methodological review for a PractiTest user (we provide this service for our customers on-demand, where we work with them to improve their testing & development process), when I noticed that not all his projects where progressing according to the flow we had defined together.

When I asked him why?  He replied:
“There is nothing I can do about it!  Sometimes programmers will simply take the code and push it to production even when I tell them that we should test it first.  This is just the way it is around here…”

What does he mean “nothing he can do” or “the way it is around here”?!?!
Isn’t he part of the team??
What’s going on here?!?!?!

The truth is that I was not really surprised.  I’ve seen it many times before and I guess I will see again in the future.  Sometimes testers feel overwhelmed and second-best to their development peers and are not willing to take a stand to defend the point they think is right.

Make your point strongly and intelligently

There is a saying in Hebrew: “It’s better to be Smart than be Right“.

I personally think a tester should try to be both (smart and right!), and in order to do this she needs to understand how to communicate and what arguments to provide whenever she wants to push one of her decisions with the rest of the development team.

Following are some situations I run into all the time, and how you as a tester can sale your point of view more effectively to your development peers:

1. When you think a bug should be fixed

When you think a bug should be fixed and not postponed to a later version you should approach the argument from a customer / market perspective.  Try to think of specific users (or user types) and how “you know” they will find this bug unacceptable.  You should try to use an argument such as “Bank ABC uses this feature extensively, and they will run into the bug at least 4 to 5 times a day”.

You can also use the argument of “a similar issue” that was reported from the field in the past and was fixed as a patch (instead of waiting for the next version).  You can then say for example: “Remember the bug XYZ we had 6 months ago, that was detected by Customer ABC, and we fixed as part of a patch because they could not wait for a formal version to be issued”.

In both cases your argument is that users will find this issue urgent and it is better to fix it now than to release the product and then issue an emergency-patch-fix later with all the overhead, the cost and the pain that come with them.

2. When you think a test should be run (or run once again)

This is the case where you think a test should be run, even if there was no time allocated to it.  A classical example is when you think that part of your regression tests should be run on all browsers and not only on IE (for example), or when you think the load testing suite should be re-run after a specific change in the product (even if it was already run before).  In both cases you are actually saying that more time will be spent on testing even though this was not planned ahead of time.

In order to allocate time for these cases you should talk about bugs that were found (either in testing or even worst by users in production) when your team did similar changes in the past.  Again, here the alternative is to release the product without running these test cases and missing bugs that will need to be fixed after the product was released to the public.

Examples of arguments to use on these cases are:
“Last time we ran tests like this we found 4 or 5 showstoppers no one expected to find”
or
“Remember when we last changed this function the response time of the server degraded about 200%”.

Whatever your argument is, it will need to come also with the reason why you now think that the tests should be run and why you didn’t think so before.  For example you can say that “in principle I thought about running only sanity, but in light of all the bugs we found on the different configurations I now think we should test in all of them full regression” or “even we already ran load testing in the past, we are experiencing bad response time even without load and I think we should evaluate this once again”.

3. When a feature should be modified or re-written

This one is similar to a bug that needs to be fixed but focused on the functionality of your product, that in your opinion will not suit what the customer is looking for.  Just like in the case of the bugs above, your best approach is to explain it from the point of view of the user, and it will be even better if you can provide the name of a specific user who will protest the current feature as it is right now.

But how can you know these things about your users???
I said multiple times before that as testers one of our tasks is to be the Customer Advocate within the development team; the reason for this is that we are the (only?) ones who constantly evaluate the product from an end-to-end perspective and based on the requirements from our end-users.  If this is the case, you should try to be as close as possible to your users and try be in contact with them to understand how they work.

When you think a feature should be changed you will need to use arguments such as “user XYZ works like this” or “remember that user ABC asked for feature XYZ to be modified because it was doing the same thing you want it to do now…”

4. When a version is not stable enough to be released

This is a hard one; sometimes you feel that even though the due date for the release is approaching we still need to run more tests even on areas we’ve already tested.

You can call it a hunch or a gut feeling, but sometimes I also have them and I’ve learned to trust them whenever they come. The hard part here is explaining to developers and project managers why you should spend more time testing areas you already tested.

Whenever this happens I do my best not to leave it as a gut feeling, but to understand what my instincts are trying to tell me.  In most cases it is related to the risk assumptions we made in the past and how after working with the product I feel that these assumptions may be incorrect.

Whatever the reason may be, once you are able to provide an explanation of the risk involved you will see how you team is more willing to make the effort and invest in the additional testing efforts.

Try always to provide hard data but also learn when to give up

When I want my team to change their minds I can try to boldly tell them that I am right and they are wrong, but this usually won’t work.

The best approach is to come up with “hard data” and “concrete examples” that will make it difficult for them to say I am wrong.  It is even better if these examples come from your current company and project, but you can also provide examples from previous projects and even from different companies where you worked int he past.

There are 2 important points to remember in these cases – when you think your team is wrong and you are right.

1. IT IS OK TO DISAGREE WITH YOUR TEAM AND YOU SHOULD MAKE YOUR VOICE BE HEARD.  Don’t hide behind the pretext that they won’t listen.  If you talk to a team member and he doesn’t want to listen, then go to the manager and tell him what you think and why!

and

2. EVEN IF YOU THINK YOU ARE RIGHT, YOU NEED TO KNOW WHEN TO GIVE UP AND EMBRACE THE DECISION OF THE TEAM OR THE MANAGER. After all, in the same way you can be right you might also be wrong, and as a team you should make decisions together and sometimes take risks and release bugs not because you want to work blindly or harm a specific customer, but because the business is telling you to work this way.

As I said before you should try to be both RIGHT as well as SMART all the time.

Today I took “half a day off” to work on developing a feature I’ve been wanting to add to PractiTest for a long time.

It’s fun to do it once in a while; to take a break, get my hands dirty, develop something for a bit and then go back to my regular work as a test manager and solution architect.  It’s not that I don’t trust my development team, I guess I did it for the fun of it

I read somewhere that it is related to changing your focus and “going deep” on a task that is unrelated to what you do all day but still closely linked to your work and how this helps to refresh your mind by breaking out of your working routine.

Writing GOOD CODE is hard!

For me as a tester it is also a reminder of how hard development work really is, a chance to experience first-hand how non-trivial it is to write something that looks good on all browsers and works correctly in all Operating Systems.

I am sure that even though I tested my “small feature” thoroughly, once it reaches someone from my team for testing he or she will find at least one or two annoying bugs on scenarios I had not thought about.  After all, isn’t this the art of testing?

Running GOOD TESTS is also hard!

The same is true about testing, you need to give a developer an end-to-end testing task for him to understand how difficult it is to test.

I heard this from a testing colleague in another company the other day, he works on an agile team where developers sometimes get testing tasks assigned to them, but then come back to him asking “if he can re-run the tests on his spare time” since they don’t feel god enough with their testing coverage.

Is this a case of someone wanting some reassurance and validation on his work, or someone who doesn’t want to take responsibility for his task and wants someone else to do it for him?  The truth is that it is hard to tell…

Teach your developers to test!

It is important for developers to take responsibility for end-to-end testing tasks once in a while; it helps them to understand what matters to us when we do our job, and maybe even more importantly it teaches them to do their jobs better!

If a programmer experiences first hand how his code behaves on a “realistic” testing environment then he can take this experience and code accordingly next time he sits to work on developing a feature.  It’s not about making him take responsibility for testing his own code, I think he simply can’t do this, but about learning to develop while been aware about additional factors that may affect his code.

To achieve this you cannot give him a 2 hours of testing task at the end of the sprint, he needs to take responsibility for a complete testing project and let him feel and live the work of a tester, even for a little while.

Doing this will not only provide you with some additional testing resources that can be used in times when the bottle neck sits in the QA department and Developers can help out, but it will also give them tools to stop bugs from crawling into the code in the first place.

Overall a win-win for the whole team.

Guys, this is a lie! The lie we tell ourselves when we need to rationalize why we are not doing our jobs right.

As testers our job is not to be the gate-keepers standing in the way of buggy software from being released. But on the other hand we should not limit ourselves to reporting bugs and then keeping quiet in the corner while the “big boys” decide on whether the product can be released now or later. If this is the way it works in your company then the problem resides on YOU and not in the rest of the team.

Respect needs to be earned

If your team does not listen to you it might be because you have nothing interesting to say. Take the time to understand what is the value you are providing.  If all you do is write down bugs, then don’t expect them to come and ask for your opinion on the readiness product.

If on the other hand you are constantly providing feedback on the product and its functionality, doing this based on your understanding of the users and the business needs, you will see how developers and product managers alike start adding you to their decision forums and seeking your opinion constantly.

Define your job as providing information and visibility and you will be adding more value than simply reporting bugs. The amount of respect you get depends on the value you add to the team.

No one likes a “party pooper“

I’ve seen many test engineers who are happy to find bugs that end up delaying the product. They feel pride in walking to the development manager with their chest up high and telling him about the show-stopper bug that will certainly delay the product for another 2 weeks.

Sometimes there is nothing we can do about it, part of our job is to deliver the bad news to the team but there are always more than one way to do this.

As a team player you need to “want to deliver the product” and the value it will provide to your team and your users.

If you don’t want to release on time, chances are you are not going to do it. But if you work release-driven, you will proactively plan your tests with the development team in order to “attack” the high risk areas first and find the complicated bugs as early as possible allowing enough time to fix them before the release is due out.

Change your mind-set and become a team player. Understand that you are working with the programmers and not against them.

Don’t be “the boy who cried wolf”

Lastly, many tester like having all their bugs fixed (who doesn’t it!).  But we need to understand that business value comes first and so it’s OK that not all our bugs will be fixed. This is specially true for bugs that are found late in the process or bugs with low user-value.

Your job as a tester is not only to report all bugs, but also to give them a severity and to provide information that will help other team members asses your view. If you make a fight for every bug you open (or if you give high priorities to all of them), soon enough people will stop paying attention to your bugs or to the times you take a stand to defend them.

Choose your fights and concentrate on the things that matter.

The change starts with you!

Make no mistake, the first step on getting recognition starts with you. Take the time today to understand how you can add more value to your team and to your product.

In the beginning it will take time, and you may even get some strange comments from your team on the “new approach” you are taking, but if you are on the right track soon enough people will start changing their attitude towards your work and you will see the difference.

Good Luck!

What is so difficult about being a tester?
This is a legitimate question from someone who has never done any good testing or who has never seen good testing being done.  So let’s take a closer look at what a tester needs to do as part of his tasks and what does that require from him with regards to skills and knowledge.

Before he even starts the “testing stuff” a tester needs to ensure that the story or spec actually fits-in with the rest of the application from a functional perspective, this is actually a “dry test” to validate we are developing the right feature with regards to the current product context. To do this a tester needs to understand the application from end to end and also to have a clear idea of how his users work with it, in contrast to a programmer who may be able to write his features correctly without even knowing what they do in the complete scheme of the system being delivered.

Also before he writes his first test or presses any link in his application he needs to make sure the feature has been specified properly and the description provided is not ambiguous or incomplete, assuring that all the stakeholders are in sync and lowering the chances of developing product “3xA” when the user actually wanted “A-A-A“. To do this correctly a tester needs to have the proper communication skills to contact all stakeholders (internal and sometimes externally) and understand that everybody is “in the same page”.

When a tester actually gets to the point of planing and executing his tests he needs to be:
- Part Jack the Ripper to “think” like the bugs and find them wherever they hide.
- Part Sherlock Holmes to tie all the clues together and understand the (elementary and) underlying issues in the system.
- Part CSI to analyze the evidence and point at the possible suspects.
- Part Oprah Winfrey to present a problematic issue in a way that doesn’t offend anyone and allows the team to solve it and move forward
- And finally part Cirque de Soleil acrobat to juggle 7 tasks while walking a tight-rope blindfolded, and still make it look easy and natural..

All through the process the tester needs to be able to detach himself from his ego and have the professionalism and the maturity not only to be self-critical of his work but to proactively seek this criticism from every other team member allowing him to run the most effective and efficient tests. At the same time he needs to be a master politician to provide criticism to his peers without making them feel degraded or insulted.

Once the project reaches its critical dates, a tester needs to become a mix between a navy-seal an ice-cold assassin and a magician; to adapt to all terrains and perform his duty in constantly changing environments, to work under the pressure of project members who all-of-a-sudden remember that the deadlines are inflexible, and to always find a way (either by relativistic quantum mechanics or sheer magic) to fit a 3 week testing schedule into 5 working days.

And if all this wasn’t enough, many times we as testers need to have skin as thick as an elephant not to get hurt by all the accusations of the things we did or didn’t do during the project, and the memory of a gold fish to forget the nasty comments thrown our way by developers or managers while “in the heat of the moment” (even if later on they come to apologize for them).

So if you ever thought testing is easy and any person from the street could come and do the same job your testers are doing I suggest you think about this again, or even better spend 1 week working with your testing team to understand how easy it really is…

This break from writing was not intentional and it was not really a break since I was really busy with tons of PractiTest work and some additional heavy-duty chores related to caring for 2 small children during their summer vacations.   But there was also a positive side to this pause since it gave me a chance to reflect on what I’ve written on the blog so far, the subjects I’ve covered and those I haven’t gotten to yet.

I did something interesting, using “wordle” (a very cool app!) I generated a word map of the blog’s content:

Word Cloud for the QABlog

It was no surprise that the word TESTING came so strong and central, but there were also some other words that took an important place and I want to take this chance to review them.


SHARE – Sharing is (almost) always positive, but in Software Development and specially in Agile Teams it becomes one of the biggest success or failure factors.
I am currently reading “Agile Testing” by Lisa Crispin and Janet Gregory and it covers this point broadly with many reasons why sharing and collaboration are two of the most important factors for succeeding in agile processes.  By the way, the book is a great read and I recommend it even to testers who are not part of Agile Teams.


TIME – If I had 3 wishes they would be happiness for my kids, world peace, and a 32 hour day!  Time is one of our most valuable assets and so we need to learn to manage it correctly.
But managing time is not enough, we also need to learn to respect it, both our time and that of our colleagues.  For me personally the biggest time waster is context switching. When I stop what I’m doing to read mails, answer IM’s or phone calls, or even when someone walks up to me to ask “just one small question” it takes me between 5 to 10 minutes to reach the level of concentration I was before the interrupt.  If you do this 2 or 3 times an hour you end up spending half the time just getting to restart your work.  Can you relate to this?


THINK – this is a big one!
Most of us don’t really think as much as we’d like to accept, we are mostly busy reacting to what is going on around us.
To really think we need to take a time-out, breath deeply for a couple of minutes and clear our heads from all the urgent things in order to focus on the important ones (notice that most times the urgent things are not necessarily the most important ones!).  When was the last time you did this??
Whenever we take the time to THINK we use our resources better by investing them on the tasks that are really needed.  It’s a shame we don’t get to do this more often…


DEVELOPERS – these guys and galls are our biggest allies, and yet they’re also the ones with whom we tend to be most in-conflict during our projects.
Our relationship with developers makes me think about the classic book “Men are from Mars & Women are from Venus” by John Grey.  In fact we need to understand that the issues between developers and testers are mainly linked to the fact that we are 2 different species (or at the least belong to 2 different cultures) and in order to communicate and work together we need to understand and respect the principles of the other side, what’s important for each of us, and how each one approaches problems and challenges in his own way.
The key lies in understanding & communicating with your colleagues- just like in all types of relationships.


PERSPECTIVE – one of my personal favorites and closely related to thinking.
When you are in the middle of the forest it becomes too hard to look at the trees.  Perspective allows us to take a new look at the issues we are working on and check for new and interesting stuff even in the places we’ve already visited multiple times before.
Gaining perspective is relatively easy, you just need to fully focus your attention on another subject for some time and then revisit your previous task.  Just by switching context you will be able to see things under a different light.
I use this all the time: when testing to make sure I really covered all the important scenarios; when trying to solve problems by looking for different approaches that may give a better result; and even when in I find myself arguing with a colleague when I take time-off to cool down and think about the stuff that is really important.  Give it a try, and tell me if it worked for you!


Lastly an interesting pairing of words that came out of the random image:
GOOD SCENARIOS – something I have not talked about much in the blog but a subject I think I will write about in the near future


I am hoping that now that summer vacations are reaching their end, and after having released some pretty amazing stuff in PractiTest that we were working on for a number of weeks, I will have some more time in my hands to continue posting more regularly…

[ NOTE: By "test" in the title I aim at the broader meaning of testing that includes all the activities around learning, planning, exploring, reporting, and obviously checking the application in order to understand if it works properly and/or find bugs on it. In this sense you could easily replace TEST with WORK and it would still mean the same.]


Last week someone sent me a link to a video by Jason Fried, co-founder of 37Signals, called “Why You Can’t Work at Work“. I’ve followed Jason’s writings for a while and most of the time I agree with his comments and ideas, but this specific short video really hit the nail for me.

What do you do with your time in the office?

So I did a short back-of-the-envelop calculation of a “typical tester workday” and I came up with the following time distribution (that I am guessing you can relate to it too):

TOTAL TIME IN THE OFFICE   ~   10 hours a day

- Morning setup (small-talk with peers, coffee & cereal)   ~   15 min
- Morning emails, news & blogs   ~   45 min
- Meetings (avg of 3 meetings  a day)   ~   3 hours
- Lunch   ~   1 hour
- Time helping others in their tasks   ~   1 hour
- Working distractions (mails, calls, “can you come for a sec…“)   ~   1 hour
- Other distractions during the day (calls, coffee, bathroom breaks)   ~   1 hour
Total Non-Working Time During The Day   ~   8 hours

ACTUAL WORKING (TESTING) TIME   ~   2 hours a day

What can you do about it?

As with most problems in life, I think the first step towards your solution is to realize you have a problem…  Then the second step is to start treating your time as something with value (to you if not to all the Organization).

What does this mean in practice?  Look for the things wasting your time and try to eliminate them or at least reduce them from your workday.

Let’s take meetings as an example:
- Don’t accept every meeting you are invited to, make sure there is a reason for you to attend (sometimes you can be more effective by simply reading the meeting summary or reviewing a document off-line)
- Demand that meetings be kept short! Who said a meeting should be 1 hour long? Why not 30 min?? Why not 20 min???
- If you feel you are not adding or getting any benefit from the meeting be polite but assertive and excuse yourself from the session.

And there are plenty of things you can do around meetings and many other time-wasters once you understand that time is your most valuable asset.

Set time aside to Do Work!

If your main problem is that you don’t find time to work, then schedule specific time-boxes in your agenda where you set aside time to work.  Try to do this when you are most productive; for me this is during the morning until around 11:00 and then from 3 to 5 in the afternoon, but this is usually different for everyone.

You need to make sure to communicate to the rest of the world that you don’t want to be bothered or interrupted during your work-time.  A good way of doing this is by posting a sign like this one in your door or desk:
If you work on an open-space or in a room with many people get a good set of head-phones (sometimes your employer may even agree to pay for them!) and find the music that let’s you concentrate on your tasks and work.

Let people know when you can be interrupted

This is a tricky one but if you want people to leave you alone to work, you also need to make sure to leave time aside to be interrupted or consulted.

For example, if you take breaks every hour (like I do) try to make them “on-the-hour” and let people know that they have 5 minutes at the beginning of each hour when they can come to ask questions.

You also need to work with non-interrupt communication mechanisms.  We use email for things that are not urgent, or IM for more urgent stuff.  I make sure that whenever I take my on-the-hour-break I check all the IM messages I had up to now, and at least 4 to 5 time a day I scan (not review!) my mailbox to see if there is something I need to attend to.

BTW, there are many people out-there who think you should go over your email once or maximum two times a day, I find this a little hard to do…

Work with the correct methodology and the tools that will help you be efficient

Email is a good example of this, when you ask co-workers to send you non-urgent requests via email instead of coming to bother you “live”.  But when you are talking directly about testing this becomes more a specific tool & method question.

Make sure your bugs are correctly written so that developers won’t come asking unnecessary questions.  All the information should be in the bug report itself!

Work with a system where your developers can automatically see the test and the steps you ran to find and reproduce the bug (I can think of at least one QA Management tool for this )

Execute your tests and give visibility to your manager so that she has all the information accessible to her all the time, don’t make her come to you seeking answers every time someone asks  a question about the test cycle or status of the product.  In short, make sure the people who depend on your work to do theirs can get all their information from the supporting testing tools and don’t require you to provide them information or even worst to interpret the information you already gave.

Telecommute

Many places don’t allow you to work off-site, but if you can get your manager to accept try to telecommute once every 2 to 4 weeks.  The amount of uninterrupted work you can do from work or from a  public library is amazing.  I also think that changing the work-atmosphere once in a while can trigger great work results.

In short…

Treat your time and your work as a valuable asset.  Once you respect it yourself it will be easier to ask others to do the same.

It all started while talking about parenting…

I had a short chat with James Bach the other day.  We didn’t talk about testing, certifications or software development; we talked about the challenges of being a good father, a more challenging task than any software project I’ve ever seen.

After that I tweeted with James and he sent me a very encouraging message saying (don’t remember the exact words but along the lines of) that he thought I would be a good father since I was self-critical.

The term SELF-CRITICAL got me thinking.
I started by analyzing how my self-criticism has helped me in almost every aspect of my adult and professional life.  Then I looked at the people around me, and realized that I was (and still am) able to work and communicate better with those in my environment who are able to auto-evaluate themselves and accept criticism more openly.

My conclusion was that the most valuable asset a self-critical person brings into a relationship is his ability to improve it by dynamically learning from the wins and losses, and evolving as the circumstances continue to change all around us.

What does this has to do with Testing?!

Well, the trivial answer is that it has EVERYTHING to do with testing!
The tester is the player in the development team that needs this quality more than anyone else.  The main reasons for this being that:

1.  Testing is a constant-learning activity.  As you run your tests, you analyze the result and evaluate them in light of your previous assumptions, in order to modify and even re-determine your testing path(/s).  The only way to do this effectively is if you are constantly open to self learning and ready to throw away your previous assumptions based on the feedback you get.

and

2.  You will never be able to provide good and deep criticism to the people around you if you are not able to accept such feedback yourself.

The most brilliant testers I’ve worked with share at least one common trait: they’re happy to learn from their mistakes, and are always willing to learn from what others tell them about what they were doing wrong and how to do it better.  Once you start accepting criticism freely you develop the ability not only to give and accept it from others, but also to give it to yourself and get it by constantly questioning if you are doing the right things.

So what is self-criticism?

It is the realization you are not expected to be perfect, since apparently no one is.
It is also the “permission” to make mistakes and errors in fair judgment, as long as you are willing to learn from them in order to make a better future.

It doesn’t mean that there are no consequences to your mistakes, but that these consequences don’t mean you cannot correct your actions the next time you have a chance to do it right.

Going back to raising kids, self-criticism might be the most important instinct we posses and one of the things we need to help develop in our kids; it is the principle that allows us to learn to walk by letting go, falling down and trying it again, realizing that falling-down doesn’t really matter in the long run.