PractiTest QA Blog

There have been multiple debates in places such as QAForums about whether Process and Project Metrics can do more harm than good to the Organization.

My view is that Metrics are primarily good; and most problems come from the way we analyze and display the information.
We also tend to forget that metrics are only an Information Channel and not a product by themselves.

Following are some Rules of Thumb I use when creating a metrics system for an Organization:

(1) Remember that metrics will always be a base for comparisons.
If you know that people will compare the metrics you provide, start by providing the base for comparison yourself.
In some companies this means to provide present numbers vs. past numbers and even vs. target numbers; in other companies this also means to provide information for other teams in the Organization.
Just don’t leave this up to the reader himself…

(2) Metrics need to be balanced.
You cannot measure and show only side of process. Make sure that you display information for all or most activities taking place in the team.
For example, you cannot provide only the number of opened bugs, but you need also to show numbers for Run Tests, Written Tests, and any other activities that measure the activities of the Testing Team.

(3) Metrics need to be normalized.
Make sure you are not trying to balance the outputs of a 15-tester team with that of a 3-tester team.

(4) Metrics need to be comparable.
Following normalization, you need to make sure you are not comparing Apples & Oranges.
Don’t try to compare a team working on an established Enterprise Application with that of a Start-Up team working on a Prototype for a Customer Oriented product.

(5) Don’t make Metrics personal.
Don’t make it a witch-hunt for a single person or a single team to take the blame.
NEVER display metrics for a single individual.
You also need to show the information in a way that the Stakeholders see the truth being displayed and not a way of trying to put blame on subjects.

(6) Think of what you want to communicate by publishing your metrics.
Don’t provide dry numbers only, specially not when you are showing possible problems that need to be corrected.
If you know that stakeholders will immediately ask questions such as “how did this happen?” or “what are we doing to correct this?”, make sure such questions are immediately answered in your report with corrective actions from the team.

(7) Metrics evolve.
It is legitimate and even necessary to review what you are measuring once in a while to make sure you are answering the needs of your organization; if the company’s priorities change from time to time so should the parameters being measured.

Most importantly, metrics are only a COMMUNICATION CHANNEL and not an objective by themselves.
They should provide correct and complete information from one group of Stakeholders to the other in order to allow for the correct decisions to be made.

Remember that you need to serve both sides of the communication process.
The best way of doing this is by not taking any sides, and by making sure both sides know this…

I was introduced to the acronym MBWA (Management by Walking Around) by a colleague who “explained to me” that this is what I did with my teams.
Silly me, I thought that this was called “management”… but I guess I was wrong )

In any case, I noticed lately that just as there are Managers who practice MBWA and those who think it is not necessary (or as one put it, overrated!), there are also those who practice MBWAYB&T and those who don’t.

Management by Walking Around Your Bugs and Tests is as simple and important as the original MBWA. The idea is to be connected to what happens in your projects at all levels, personal as well as professional, and keeping up to date not only with the statistics but also with the actual tests executed and issues being detected.

For me, the advantages of doing this are multiple:

1. I am able to provide proactive and direct feedback to the people who report to me and their teams.
This also helps them feel I am part of the on-going process and assign enough importance to what they do in order to review it.

2. It gives me a good visibility into what is going on in my projects.
This specially important during crunch times, when we need to make quick and risky decisions on the spot.

3. It gives me a good indication of the real and low level needs of my teams.
I am able to help my team leaders to handle things that might not be obvious to them, and thus help them and their teams.

4. All the “cliches”.
It keeps my testing senses up to speed.
It helps me to learn the new features.
I feel part of the team.
etc…
(not untrue, but in some ways trivial)

There are also potential problems with this practice that should not be overlooked.

1. I’ve seen Managers that walk around so much they don’t get to do anything else. In fact it takes a lot of discipline to make this a routine that is both effective and efficient.
You need to limit the amount of time you do this and do it on a regular basis, that way it is easier for you only to go over the stuff that changed or was added since the last time you checked your information.

2. Your employees and specially the managers who report to you need to understand what you are doing and why. If they think you are micromanaging (or even worst, that you don’t trust them!) they may feel threatened by your actions.

3. You should not do the job of your employees or your managers, make sure you know where to draw the line.
There are times you need to let people do what they think is correct even if you think otherwise; and at times you’ll need to let people fail in order to learn their lessons.
Be smart and know when to Let Go!

I know that some managers say that the amount of data reported by their teams does not allow them to do this.
I DISAGREE, and I tell them that even if they check only the major bugs and only a sample of the tests that fail they will gain more than by going over their daily numbers when they arrive and before they leave work…

It may only be me, but I don’t feel I am managing my team unless I know how they feel and what they are doing. This is why I practice and encourage MBWA and MBWAYB&T .

Most if not all development projects are short on time, resources and tests. This means that we as QA Managers and Engineers will be asked to cut our testing operations in order to accommodate for the delays of all the other project teams.

So how can we manage to test less and still deliver our obligations?
We cannot count on Testing Muscle or Strength, we will need to rely on intelligent testing & prioritization…

We can do this by correctly defining of our Jobs as providing Testing Intelligence™:
Relevant and timely information, captured from multiple sources and using many disciplines, that will help the project stakeholders make their tactical and strategic decisions.

Based on this definition, I see testing only as a tool and not the objective.
I already wrote, and I still stand by it, that if I could provide the same information without running a single test I would still be doing my Job right.

Reporting the bugs in the system is also important, but bugs are the responsibility of the developers who wrote them into the product and our job is to help by detecting the most important defects and focusing developers on the areas where they need to improve their code.

So if our main task is to provide information, we should ask ourselves 2 things:

1. Who are the stakeholders who need this information?
and
2. What information does each stakeholder need to make her/his tactical and strategic decisions?

Only you can answer these questions since they are directly related to the nature of the product, the team, the end-users, and many additional constraints that define you project.

Once you answer them, you will be able to change your mind-set and plan your testing process based on the information you’ll need to provide and the data needed for it.

Maybe most importantly, you will be able to decide how to reshape you test plans when they need some trimming or remolding.

An additional advantage of working this way is that you will understand your Internal Customers and their needs better. You will see that they need specific communication channels to receive their information in the most efficient way (Clue: this is not always as a 10-page document full of tables and graphs).

But I will talk about the different information channels on a separate article, this one is big enough as it is…

This is one of the most charged discussions I’ve been in, and what makes it more annoying is the fact that it usually takes a political taste in many Organizations. In any case, I think I managed to understand the confusion and find my answer…

I was at SIGiST meeting this week in Tel Aviv, and among other things there was a presentation by a QA Manager from SAP who showed the Automation Harness they had developed in his team. Just a few words about the platform, it is pretty amazing (and this from the perspective of having managed the QA for QTP and QC in the past, and from working on PractiTest now).

In any case, in his introductory slide he said that he sees himself as a Test Manager and not a QA Manager, but that this was his formal title in SAP.
At the beginning it got me all fired up inside, since I do believe that in all the Organizations I worked so far I functioned as a QA Manager, and here was yet another person who limited our task and responsibility only to test execution.

Then as his presentation continued, I understood that he was right.
HE really is a Test Manager, and his responsibility is to create and facilitate the tests that other people use and run in order to test their software. He does not perform the extra task of running the tests as part of a strategically designed project that provides visibility into the status of the AUT and its development process.

Eureka!
Apparently there are Test Managers, I am simply not one of them since I am also in charge of the QA.

For me testing is only a tool. I use it in order to fulfill my objective of providing visibility. This does not mean that I am solely in charge of the Quality of the Product, this is a task & responsibility shared by all the people involved in the development process; I am the Stakeholder in charge of understanding and communicating the Quality Level achieved by the AUT to my peers.

So here is the end of a very long question for me.

I am a QA Manager.