This break from writing was not intentional and it was not really a break since I was really busy with tons of PractiTest work and some additional heavy-duty chores related to caring for 2 small children during their summer vacations.   But there was also a positive side to this pause since it gave me a chance to reflect on what I’ve written on the blog so far, the subjects I’ve covered and those I haven’t gotten to yet.

I did something interesting, using “wordle” (a very cool app!) I generated a word map of the blog’s content:

Word Cloud for the QABlog

It was no surprise that the word TESTING came so strong and central, but there were also some other words that took an important place and I want to take this chance to review them.


SHARE – Sharing is (almost) always positive, but in Software Development and specially in Agile Teams it becomes one of the biggest success or failure factors.
I am currently reading “Agile Testing” by Lisa Crispin and Janet Gregory and it covers this point broadly with many reasons why sharing and collaboration are two of the most important factors for succeeding in agile processes.  By the way, the book is a great read and I recommend it even to testers who are not part of Agile Teams.


TIME – If I had 3 wishes they would be happiness for my kids, world peace, and a 32 hour day!  Time is one of our most valuable assets and so we need to learn to manage it correctly.
But managing time is not enough, we also need to learn to respect it, both our time and that of our colleagues.  For me personally the biggest time waster is context switching. When I stop what I’m doing to read mails, answer IM’s or phone calls, or even when someone walks up to me to ask “just one small question” it takes me between 5 to 10 minutes to reach the level of concentration I was before the interrupt.  If you do this 2 or 3 times an hour you end up spending half the time just getting to restart your work.  Can you relate to this?


THINK – this is a big one!
Most of us don’t really think as much as we’d like to accept, we are mostly busy reacting to what is going on around us.
To really think we need to take a time-out, breath deeply for a couple of minutes and clear our heads from all the urgent things in order to focus on the important ones (notice that most times the urgent things are not necessarily the most important ones!).  When was the last time you did this??
Whenever we take the time to THINK we use our resources better by investing them on the tasks that are really needed.  It’s a shame we don’t get to do this more often…


DEVELOPERS – these guys and galls are our biggest allies, and yet they’re also the ones with whom we tend to be most in-conflict during our projects.
Our relationship with developers makes me think about the classic book “Men are from Mars & Women are from Venus” by John Grey.  In fact we need to understand that the issues between developers and testers are mainly linked to the fact that we are 2 different species (or at the least belong to 2 different cultures) and in order to communicate and work together we need to understand and respect the principles of the other side, what’s important for each of us, and how each one approaches problems and challenges in his own way.
The key lies in understanding & communicating with your colleagues- just like in all types of relationships.


PERSPECTIVE – one of my personal favorites and closely related to thinking.
When you are in the middle of the forest it becomes too hard to look at the trees.  Perspective allows us to take a new look at the issues we are working on and check for new and interesting stuff even in the places we’ve already visited multiple times before.
Gaining perspective is relatively easy, you just need to fully focus your attention on another subject for some time and then revisit your previous task.  Just by switching context you will be able to see things under a different light.
I use this all the time: when testing to make sure I really covered all the important scenarios; when trying to solve problems by looking for different approaches that may give a better result; and even when in I find myself arguing with a colleague when I take time-off to cool down and think about the stuff that is really important.  Give it a try, and tell me if it worked for you!


Lastly an interesting pairing of words that came out of the random image:
GOOD SCENARIOS – something I have not talked about much in the blog but a subject I think I will write about in the near future


I am hoping that now that summer vacations are reaching their end, and after having released some pretty amazing stuff in PractiTest that we were working on for a number of weeks, I will have some more time in my hands to continue posting more regularly…

Related posts:

1. When you think you’ve done enough, go for a walk and then think again
2. Testing the image in the mirror
   - or -
   Why a good tester needs to be Self-Critical?

It all started while talking about parenting…

I had a short chat with James Bach the other day.  We didn’t talk about testing, certifications or software development; we talked about the challenges of being a good father, a more challenging task than any software project I’ve ever seen.

After that I tweeted with James and he sent me a very encouraging message saying (don’t remember the exact words but along the lines of) that he thought I would be a good father since I was self-critical.

The term SELF-CRITICAL got me thinking.
I started by analyzing how my self-criticism has helped me in almost every aspect of my adult and professional life.  Then I looked at the people around me, and realized that I was (and still am) able to work and communicate better with those in my environment who are able to auto-evaluate themselves and accept criticism more openly.

My conclusion was that the most valuable asset a self-critical person brings into a relationship is his ability to improve it by dynamically learning from the wins and losses, and evolving as the circumstances continue to change all around us.

What does this has to do with Testing?!

Well, the trivial answer is that it has EVERYTHING to do with testing!
The tester is the player in the development team that needs this quality more than anyone else.  The main reasons for this being that:

1.  Testing is a constant-learning activity.  As you run your tests, you analyze the result and evaluate them in light of your previous assumptions, in order to modify and even re-determine your testing path(/s).  The only way to do this effectively is if you are constantly open to self learning and ready to throw away your previous assumptions based on the feedback you get.

and

2.  You will never be able to provide good and deep criticism to the people around you if you are not able to accept such feedback yourself.

The most brilliant testers I’ve worked with share at least one common trait: they’re happy to learn from their mistakes, and are always willing to learn from what others tell them about what they were doing wrong and how to do it better.  Once you start accepting criticism freely you develop the ability not only to give and accept it from others, but also to give it to yourself and get it by constantly questioning if you are doing the right things.

So what is self-criticism?

It is the realization you are not expected to be perfect, since apparently no one is.
It is also the “permission” to make mistakes and errors in fair judgment, as long as you are willing to learn from them in order to make a better future.

It doesn’t mean that there are no consequences to your mistakes, but that these consequences don’t mean you cannot correct your actions the next time you have a chance to do it right.

Going back to raising kids, self-criticism might be the most important instinct we posses and one of the things we need to help develop in our kids; it is the principle that allows us to learn to walk by letting go, falling down and trying it again, realizing that falling-down doesn’t really matter in the long run.

Related posts:

1. A chance to take a look back
2. My Top 6 Tester Qualities during a Job Interview

I read a blog by James Whittaker analyzing how for him Software Testing is still stuck in the 90s while the rest of the (technological) world has evolved greatly.  Today I will disagree with Mr. Whittaker since I think we as testers have not thrown all we’ve done away and we have seen real advancement in many areas of our work field.

As he writes in his post, back in 1990 I was a 16 year old high-school student, and as he says I didn’t have an email address (I will say that even in Costa Rica I did have a PC computer running Windows).  To tell the truth I only started doing software testing around 1997, but I still have a clear reference point and can definitely see how the Testing Profession has evolved in these last 10-13 years.

Methodological Testing

We can talk about important methodological advances with the implementation and proliferation of testing techniques such as ET and even some more complex techniques that are still been developed and perfected such as Model Based Testing.

As a tester and a test manager I have seen how we evolved from working as an ad-hoc,  unorganized and (at least in my mind) unprofessional bunch of “button-clickers”; and became professionals in the areas of risk analysis, user advocacy and both structured and heuristically testing techniques.

Not to mention what the current Agile (r)evolution is doing not only to the Development world in general, but to the testing profession in particular.

Testing Tools

When we look back at the subject of the tools we use in testing we can see that many of the “magical-record-replay” solutions that tool makers tried to sell to us felt short from the empty promises made on demos & presentations.

But we have also seen how these same functional automation tools evolved, and how the object recognition approach they were based on became very useful technology.   If you follow this area closely you can see how new approaches such as Keyword Driven Testing provide a more robust solution (still far from perfect), and how we have expanded the reach of the automated testing realm to include more and more business experts to collaborate with our day-to-day work.  Additionally we can see how today we have excellent free tools such as Selenium or Watir (to name only 2!) that make this area more economically accessible to all.

Test Management has also evolved from a science based on excel-sheets and emails, to advanced QA Management Platforms that allow us to seamlessly manage teams and processes regardless if they are all located in a single room, or dispersed over the world in 3 continents and separated by 15 time-zones.

And finally we can look at the advances in load testing tools with the low level analysis that can be reached today with some of the most advanced tools out there, allowing us not only to determine bottle-necks but show the specific areas in the system where they occur.  This is far from the load testing results of “500 users it works – 510 it crashes” that we were able to provide 10 years ago.

The Testing Career & Global Community

Lastly, and to cut it short since we can continue providing examples for a long time we can also talk about the career of the software tester.

What used to be a 1-2 year span before you became a real developer, product manager, or support engineer; became a legitimate career for many excellent professionals who see themselves growing and developing as testers.

Just the increasing amount of professional paths you can take within testing, each with its specialization and challenges, make testing more and more attractive for many engineers that either enter testing directly from school or get into testing after spending time doing development and looking for a more interesting and challenging career path.

Add to this the expanding online tester communities (for example the SoftwareTestingClub) that let you connect, share and learn from many great professionals around the globe and you get additional positive momentum that is already taking our profession higher and higher, day after day!

Maybe its time to turn on the lights

I have no doubt that the overall value we testers generate today is a lot more than that we did 10, 15 or 20 years ago.

So, I think that many of the people who still think we are stuck in the 90s and that most of what we’ve done has been spent and thrown away without really advancing our profession or the way we work today, are suffering from some sort of myopia or lack of perspective.  Maybe a good way for them to come out of the dark ages of testing would be simply by turning on the lights and taking a good look around.

Related posts:

1. Coffee, Nature & Software Testing…

This saying is also very accurate in the case of testing, where if you do a test that finds critical issues but then you fail to report them it is as if you had not done a thing (or worst!).

So the trivial lesson is: Find a bug, make sure you report it.

But as Test Managers our problem is more complex than this.  At times, even if we work right and report all our findings to the rest of the team, we see them disregarding our warnings and acting as if we had not said anything to them (e.g. choosing not to fix critical issues, or  releasing a product that is not ready for the market).

We can try to blame the rest of the Organization for failing to listen, but this would be “taking the easy way out”.  I think that most of the times when the rest of the team fails to understand what we are trying to say it is because we didn’t communicate correctly with them.  Or as a friend of mine once said, because our message was “clear as mud“.

Thinking about your audience before writing

Providing information is about telling people what they need to know, when they need to know it, and in a way that will be useful to them.  We need to think about our readers, in this case the rest of the Development or Management Team, and understand what information they need and in what format it will be useful to them.

Many times we write reports as if they were intended to be read by the QA team only.  They include information in a format and style that may not be easy to understand by people outside our team, or that can even be alienating for non-technical people.  In many cases we simply write too much information that is not useful to anyone!

Our external documents and reports need to be formatted based on the needs and customs of our readers, and include the important information in a way that will make immediate sense to them, and will allow them to make decisions quickly and easily without the need to decipher, analyze or in some cases even translate the information from our internal jargon to the non-techie language they understand.

Remember that many times, and specially in written reports, Form is as Important as Content.

Deciding what information should not go into the report

As I started writing above, one of the most important things is to decide what information to include and what to leave out of your reports and documents. Your audience (the people you want to read your stuff!) have limited time and they expect you to choose what information is relevant for their decisions.

People expect you to to understand what stuff is important and what is secondary to the decisions being made.  If you are not sure about this you can consult with one or two members of your audience, and try to understand not only what is important in your current report but what are the principles that make something important so that you won’t need to consult with them all the time.

Value your audience’s time and they will value your opinion.

When to speak up and when to keep quiet

A critical aspect when taking part of a meeting or review process is to decide when to speak up and when to keep quiet.  What you are trying to avoid here is to be taken as the guy (or girl) who is always bringing forward the negatives points on each decision, and when there are not negatives points to make them up just for sport.

Even if your task is to bring forward the risks, bugs and drawbacks you need to make sure to point them out only when relevant and not to fall under the definition of “the boy who cried wolf” in your team.

Provide Information and Stop Being the Gate-Keeper

To summarize, we need to keep in mind that our Job as Testers is to provide the information that will allow the Management Team to make the right decisions.

Management are not very interested in our tests, mostly in our information.  These guys want to know whether the product can be released.  If yes with what risks or potential bugs, and if no why and what needs to be achieved (or fixed) before we can do it.

Related posts:

1. Testing Outputs that generate added value to the Organization
2. The art of transforming testing data into project information
3. Pair-Testing a practical solution for getting out of the mud

The conference started yesterday with some tutorials in the morning and presentations in the afternoon, where among other things I presented the concept of Testing Intelligence, with good comments and approvals from the session participants.

I was happy to see that concepts and approaches similar to Testing Intelligence are been developed in parallel by multiple QA Specialist who, same as myself, are realizing that Testing is not only about coverage, bugs, and AUT’s; but about information, stakeholders, and working within a team focused on developing a product.
One of these people is Derk-Jan de Grood from the The Netherlands who gave a great tutorial about Results Driven Testing, with some parallel lines of thought to the ones around Testing Intelligence.

I’m looking forward to a couple of more days of great sessions, starting today with a keynote from Mary Poppendieck…  Will post more in the following days, so stay tuned

Related posts:

1. Testing Intelligence™ – your deliverable and product as a QA Profesional
2. Coffee, Nature & Software Testing…
3. Testing Intelligence, a case for Intelligent Testing

I’m actually in Costa Rica, as some of you might know I was born & raised here so this is kind of a friends & family trip but not only that.
The last time I was here a friend commented about the growing local software development industry and this got me thinking, so a couple of months ago I started doing a little research and decided to put together a half-day seminar around Testing Intelligence and some additional topics making waves in the world of testing today.

The “event” took place this last Friday and I was really surprised to see the amount of people who showed up.  We got to fill the conference room with close to 50 people (Testers, Managers, Developers, Analysts, etc) from about 20 different companies (Banks, Development Firms, Outsourcing Companies, Consultants, etc) who were all interested in learning more about how to develop their products more effectively by applying their Tests in a more focused and coordinated way.

For me it was a blast!  To be able to talk about Testing Intelligence and other testing topics with people from Costa Rica was really great; specially to understand that here, in a country that up to now I had linked only to Great Coffee and Breath-Taking Nature there is also a growing community of testers who share the same dilemmas, challenges and issues that I get to see all over the world.

I am really hopping to continue contributing and working with this testing community to help it grow and expand.  As I told them during the session, they are keeping awfully quite and they need open-up to online communities and all the additional sources of information around the Internet…

So here goes for Costa Rica!  Here goes for excellent coffee, for green tracks along the volcanoes & beaches, and for a growing community of IT professionals (and specially their testers!).

Lastly I wanted to thank all the people who helped out on the organization of the seminar, and specially to Melissa Castillo who quietly and firmly made everything happen smoothly.  THANKS!!!

Related posts:

1. Do you think software testing is still in the Dark Ages? Turn on the lights!!!
2. Testing Intelligence in QA&Test ’09 – Bilbao
3. What Software Testing can learn from Basketball, games are won with a Great Defense

Who hasn’t gone through this before?
I’ve certainly been in the same situation enough times myself, and over the years I’ve learned some tricks and developed a couple of techniques that help me avoid such situations, or at least reduce their number significantly.

1.  I make my homework before the meetings
I always try to talk about the important issues with the project stakeholders One-on-One before the actual meeting where the decision will take place.  This way I can spend the necessary time to thoroughly explain the issues and go over the repercussions it may have for each person and team individually; this also gives them time to think about the issue quietly and ask additional questions if necessary.
If possible I also try to understand from the people I meet what’s their position and thus I know what’s my status before going to the meeting itself.

2.  I make my objective to become the Trusted Adviser of my project Heavy-Weights
This one is easier said than done, but it is the most effective method.
In principle it means that you need to create a reputation for yourself that will make people listen whenever you speak your mind.
How to do this is a science by itself, and it’s definitely not something that happens overnight.
Personally I try achieving this using the principles of Testing Intelligence: “By providing correct and timely visibility into the product and process, that helps my Organization’s Stakeholders make the correct tactical and strategic decisions.”
An additional good approach is also to map your project heavy-weights (the stakeholders who usually carry the most weight in the important decisions) and start by becoming their trusted advisers, as soon as the other stakeholders realize these guys are listening to you they will start doing so themselves.

3. I’m open to change my mind when I understand new things about the problem at hand
Don’t be a hard-head and learn to change your mind, this is the reason I make a point of really listening to the other side when I’m arguing.  One of the dumbest mistakes you can make is to continue fighting for an argument even when everyone realizes you are not right.
The most common cases are when someone presents a new argument you were not aware off (e.g. customer pressure, market reality, etc) and you need to understand that the good of the Company and the Product require the team to take the path you did not support initially.
Smart people need to be smart enough to know when to change their minds…

4. I learned to pick my fights
Since I cannot fight over every decision, I make sure to choose what things are worth fighting for and what battles I need to conceive in order to concentrate on the important stuff.

5.  Learn how to communicate
Maybe the most important, and yet one of the hardest to things to achieve.  We all need to learn how to transmit our messages clearly to the other side.
Just to show a good example of how to do this I will point to another article by Rob Lambert that illustrates one of these techniques (and no, I am not getting any royalties from Rob today…).

To summarize, there are many things you can do in order to confront and even succeed on these kind of situations, and all of them start by recognizing you have additional human beings in front of you.
Try to think what you can do in order to help them understand your argument and your point of view, then you will have no choice but to let them make their own decisions.

Related posts:

1. When to break down your QA Management Project into multiple smaller projects
2. Release Criteria – defining up-front when the product will be ready to be released
3. Testing Intelligence, a case for Intelligent Testing

My view is that Metrics are primarily good; and most problems come from the way we analyze and display the information.
We also tend to forget that metrics are only an Information Channel and not a product by themselves.

Following are some Rules of Thumb I use when creating a metrics system for an Organization:

(1) Remember that metrics will always be a base for comparisons.
If you know that people will compare the metrics you provide, start by providing the base for comparison yourself.
In some companies this means to provide present numbers vs. past numbers and even vs. target numbers; in other companies this also means to provide information for other teams in the Organization.
Just don’t leave this up to the reader himself…

(2) Metrics need to be balanced.
You cannot measure and show only side of process. Make sure that you display information for all or most activities taking place in the team.
For example, you cannot provide only the number of opened bugs, but you need also to show numbers for Run Tests, Written Tests, and any other activities that measure the activities of the Testing Team.

(3) Metrics need to be normalized.
Make sure you are not trying to balance the outputs of a 15-tester team with that of a 3-tester team.

(4) Metrics need to be comparable.
Following normalization, you need to make sure you are not comparing Apples & Oranges.
Don’t try to compare a team working on an established Enterprise Application with that of a Start-Up team working on a Prototype for a Customer Oriented product.

(5) Don’t make Metrics personal.
Don’t make it a witch-hunt for a single person or a single team to take the blame.
NEVER display metrics for a single individual.
You also need to show the information in a way that the Stakeholders see the truth being displayed and not a way of trying to put blame on subjects.

(6) Think of what you want to communicate by publishing your metrics.
Don’t provide dry numbers only, specially not when you are showing possible problems that need to be corrected.
If you know that stakeholders will immediately ask questions such as “how did this happen?” or “what are we doing to correct this?”, make sure such questions are immediately answered in your report with corrective actions from the team.

(7) Metrics evolve.
It is legitimate and even necessary to review what you are measuring once in a while to make sure you are answering the needs of your organization; if the company’s priorities change from time to time so should the parameters being measured.

Most importantly, metrics are only a COMMUNICATION CHANNEL and not an objective by themselves.
They should provide correct and complete information from one group of Stakeholders to the other in order to allow for the correct decisions to be made.

Remember that you need to serve both sides of the communication process.
The best way of doing this is by not taking any sides, and by making sure both sides know this…

Related posts:

1. Measuring the execution performance of a testing team
2. The art of transforming testing data into project information
3. Testing Intelligence™ – your deliverable and product as a QA Profesional

So how can we manage to test less and still deliver our obligations?
We cannot count on Testing Muscle or Strength, we will need to rely on intelligent testing & prioritization…

We can do this by correctly defining of our Jobs as providing Testing Intelligence™:
Relevant and timely information, captured from multiple sources and using many disciplines, that will help the project stakeholders make their tactical and strategic decisions.

Based on this definition, I see testing only as a tool and not the objective.
I already wrote, and I still stand by it, that if I could provide the same information without running a single test I would still be doing my Job right.

Reporting the bugs in the system is also important, but bugs are the responsibility of the developers who wrote them into the product and our job is to help by detecting the most important defects and focusing developers on the areas where they need to improve their code.

So if our main task is to provide information, we should ask ourselves 2 things:

1. Who are the stakeholders who need this information?
and
2. What information does each stakeholder need to make her/his tactical and strategic decisions?

Only you can answer these questions since they are directly related to the nature of the product, the team, the end-users, and many additional constraints that define you project.

Once you answer them, you will be able to change your mind-set and plan your testing process based on the information you’ll need to provide and the data needed for it.

Maybe most importantly, you will be able to decide how to reshape you test plans when they need some trimming or remolding.

An additional advantage of working this way is that you will understand your Internal Customers and their needs better. You will see that they need specific communication channels to receive their information in the most efficient way (Clue: this is not always as a 10-page document full of tables and graphs).

But I will talk about the different information channels on a separate article, this one is big enough as it is…

Related posts:

1. Testing Intelligence™ – your deliverable and product as a QA Profesional
2. Testing Outputs that generate added value to the Organization
3. Ask yourself what were you hired to do?

So, think about it… What are you expected to contribute in the scheme of your Product Development Project?

Are you supposed to catch all bugs before releasing the product to the world?
I think Not. It is exuberantly expensive to reach this level of bug-free-environment, and 99% of the projects don’t need it anyway.

Do you need to run as many tests as possible?
The answer again is NO! If you could get the same information (or confidence level) about the product without running a single test you would still be doing your Job – maybe even in a better way!

The real added value we provide as testers is something called Testing Intelligence™: Timely, accurate and actionable information about the status of your product and project.

In simpler terms (as I defined in a past article), we need to provide correct and timely visibility into the product and process, that will help the Organization make the correct tactical and strategic decisions.

I call it Testing Intelligence since the process and its outputs have many similarities with what is known as Military Intelligence:

1. It is based on the principles of gathering information from multiple sources, and mixing it to create a model representing the current situation and status.

For example, in order to give a complete estimation of your project you will use the data from:
- Test results showing pass vs. fail numbers and percentages
- Total number of bugs opened, fixed and closed
- Load benchmarks for your application under different levels of load
- Test execution and bug resolution trends to indicate the progress of the project and the way it is coming along based to the original plan
and
- Historical data for similar projects, to serve as a base for comparison

You will then take all this isolated data points, you will create direct and indirect correlations, and then transform it into structured information (e.g. a Report) that provides a better image of the product’s status and of the project’s chances of meetings its schedules and targets.

2. The results of your analysis cannot be considered an exact science – by you or your stakeholders.

Since you are constantly making assumptions based on incomplete data, it is normal to expect to be wrong part of the time.
Each statement provided should come with an associated level of risk (and in some cases an explanation of the source of such a risk) that will allow your stakeholders to make more balanced and informed decisions.

3. The way you understand the situation will change as you become aware and analyze new data and information.
You should not be afraid or ashamed to modify your conclusions and provide an updated model that describes the situation better, you need to explain why this happened and how it affects your previous estimations.

4. The way you present the information needs to be defined by the needs and likes of your stakeh0lders.

If they expect to see only numbers and a short conclusions don’t go around publishing long pages full with explanations. If on the other hand they like reaching their own conclusions, you should provide them a report with your data, followed by your assumptions, and ending with your conclusions; then you should work with them and let them agree or disagree with your results.

In any case the important thing is to give them what they want, but to make sure you don’t paint a picture prettier or uglier based on their personal believes or interests.

Working with incomplete information does not give you the right or an excuse to make a sloppy job. It is actually the reason why your professionalism, self-criticism, and 360˚ understanding of the situation is even more important for your work and the well-performance of your Company.

Be proud to be a Tester, your job (if done correctly) is more complex and demanding than most of the other members of your team.

Related posts:

1. Testing Intelligence, a case for Intelligent Testing
2. Release Criteria – defining up-front when the product will be ready to be released
3. Taking your product for a successful test drive – How to run a good Beta Program