PractiTest QA Blog

Retrospectives can potentially be one of the most beneficial tools in software development; then again most retrospectives I’ve participated in where a complete waste of time…

The idea of a retrospective, or postmortem as we called them about 10 years ago, is simple. At the end of the project take the time to define 3 things:
1. What went right and should be kept?
2. What went wrong and should be changed?
3. How to make things better the next time around?
So, how come most projects either don’t perform retrospectives, or carry out postmortems but don’t learn from them and keep doing things in the same unproductive way?

The answer to the first part of the question comes from the statement behind the second part. If a Manager “tries” doing retrospectives on a couple of projects and doesn’t get any added value, why should she continue doing them?
I think the problem is not with the concept of the Retrospective, but with how it is carried out in many Organizations and what is done with the results that come out of it.

There are a number of recurring mistakes and wrong-doings that can be seen in most postmortem failures:

1. Lack of Manager Leadership – If Senior Managers don’t give retrospectives their full attention and don’t communicate (using personal example) their importance, then middle management and the rest of the team will not treat these activities seriously.
Postmortems need to be lead by someone with authority and charisma; this is the only way to successfully guide the team through the complex process of reviewing, analyzing and learning from past activities and results.

2. Not gathering issues and facts throughout the project – Bad retrospectives focus mostly on the issues that happened during the last legs of the process, since these are the things we remember better and the ones that still have an emotional effect in most of us.

It is very hard to keep track of issues if we don’t record them when they happen, and it is even harder to analyze them if we don’t have objective data. The Project or Team manager needs to find ways to record important issues when they happen, and store information that will help the team understand how and why they took place.

A good idea I got from a Group several years ago was to create an e-mail alias that people could send important issues as they happened. At the end of the project the team running the retrospective can use these e-mails as a starting point for their process.

3. Letting postmortems turn into Witch Hunts – In some Young Organizations or after a very stressful project a postmortem can easily turn into a personal Witch Hunt. The most effective way to stop this from happening is by not letting it start in the first place.

If during the analysis and discussion process the team starts taking the subject into personal tones and levels, it is the job of the Manager to stop this on-the-spot and to turn the subject to a solely constructive and impersonal discussion. If necessary she should make a time-out or call a recession until people have a chance to cool down.

4. Lack of clear Lessons Learned, Action Items and Follow-through Plan – The big mistake of creating a large list of things to improve and yet continue working in the same way…Improvement doesn’t happen by itself, it needs to be pushed by people who keep track of the unwanted behavior and constantly make an effort to correct it.

A good practice is to focus on only 3 things to improve from each retrospective, and then during the following project to implement a way to measure these 3 things and constantly track their improvement.

I personally like to take my targeted improvements and start reviewing them on the weekly (or periodical) management and team meetings, 10 minutes before each meeting and 2 minutes during the meeting itself is usually enough to ensure these action items stay in-mind and on-track.

Last but not least, as I was looking more material for this article I found something really amazing from a group of people I was not taking seriously enough. The Game Development Industry, or at least the New Engineers specializing in this area have been using and publicly posting their retrospectives more than any other groups that I know of. I suggest you take the time to review this site, and learn both from the way the can seriously analyze their wins and mistakes; you might even read something that can be leveraged to your team or project.

QuackTest is in the process of sharpening its feathers in order to take its maiden Beta Flight. In fact, there are already a couple of Organizations running the platform in their production projects as part of our initial steps into this vital part of the software development lifecycle…

The Beta is one of the most important parts of the development process. It is the crucial time when the product is initially released to a limited number of customers to be used out of the controlled environment of the testing lab (or developing team environment).

Each Beta is different, in the same way that each product release is different too. For this reason you need to plan each Beta program taking into consideration its individual characteristics, objectives, and constraints.

Do you even need a Beta?

Most products can benefit from a Beta program, but not all of them need it. Home-built applications and specialized custom-built systems will benefit more from a Pilot Project, that is similar but at the same time very different from a Beta Program.

Betas are for products that will be deployed on user environments with multiple and different configurations (more than what we can effectively validate in our Testing Labs), and that will be operated by users on ways that we can seldom anticipate.

The main idea behind the Beta is to perform a “limited” release for users who understand the product has been thoroughly verified but it may still contain unknown defects that need to be found & fixed – it’s the accepted way of releasing your product with a big disclaimer sign, giving you permission to still have bugs…

Defining your Beta

Start by defining the Objectives or Goals you want to achieve. Goals usually fall into 2 categories: Feature validation and Environment (or product) verification.

Since you want to validate specific features and verify a defined set of environments you should look for an appropriate distribution of Beta Users. Even if most times you cannot choose your specific Beta Customers it is important to know what you’re looking for, and maybe perform proactive activities to help you fill in the missing spots.

Once you’ve defined your goals, you will need to review your constraints; specifically the amount of time and resources you can assign to the program.

You will need to provide special support to your users, you’ll also want to quickly process the data you get from the field in order to have a chance to affect the product prior to the release deadline, and you will also be required to handle the issues that seriously affect or harm your users. All these operations take human resources and time to handle.

Finally you will need to understand your time constraints, as for most companies (except maybe google and gmail!) a Beta is limited in time and needs to be completed in order to be able to release the product to the general public.

Running your Beta

It should already be clear from the previous section that the success or failure of the Beta mainly depends on people.

It depends on the users who will run your product and report the issues they encounter; and it also depends on the employees who will gather the issues and take care of the users during the program duration.

Pay attention to both these groups, make sure to prepare them and give them the tools necessary to perform their tasks effectively and efficiently.

Gathering Information

There are 2 ways to gather information during Beta Programs. Most companies only use the reactive way, where they open a communication channel (web-form or email account) for users to report the issues they encounter.

Surprisingly enough, companies don’t create a direct way to gather information from users. A direct channel is created when you interact with your customers and gather first hand information and experiences from their use of the product.

Direct information channels can range from face-to-face meetings to web-based questionnaires where you ask for specific information around defined aspects of the application and user experience.

Running a Beta without gathering all the possible information is like investing a lot of money on a whole set of tools but choosing to use only a small part of them.

Summary

It is important not to release a product to Beta before it is ready, it may harm the confidence of your users in you and your product.

Make sure you have a good team planning, running and handing the beta. Work with your users and help them help you to get the information you require.

As I’ve wrote a couple of times in the past, a fool with a tool is still a fool, and a Beta Program is nothing more than another tool you have at your disposal in order to assure you release the product right.

My laptop died today. It got stuck in mid work; I tried rebooting and it started displaying a consistent blue-screen with the message “Unmountable boot volume” written in bold white letters.

The annoying thing is that I saw it coming.
A couple of weeks ago I noticed that it had started getting extra hot after short amounts of time working (maybe one of the fans just went dead?), and last week I even told a friend that I was thinking of backing up my data and formatting my hard drive it since it was starting to get stuck.

Sometimes you feel thing starts going bad, but it’s only when they crash that you put all the warning signs together…

I think the same is true for many things, and I’ve seen it happen in multiple Testing & Development Projects.

I could start giving examples of how it evolves from consistent build slips and ends up with products released with large amounts of bugs and terrible overall quality; or I could talk about the cases where it starts by one tester showing late to work every day and ends up with the whole team quitting over a bad manager; whatever the case it is always a gradual process that gives multiple signs to whoever has his eyes open and is willing to see.

I won’t provide any do’s and don’ts today, my laptop died and I’m not a good mood (
Today I will only remind all of us that there are always signs on the wall; the problem (my problem!) is that we are too busy to read them when it can still make a difference (to our projects, to our teams, or to our laptops…)

I ran into someone looking for information about the CMMI. He was interested in the criteria that placed a company (or better defined, its process) under Level 3 instead of Level 2.
This reminded me of past times when I took part (and even lead) the certification processes in similar Organizations.

I don’t intend to demonize Certifications. I don’t think they are bad for the Company or its customers, and I certainly don’t claim to be smarter than the people who run the CMMI or ISO certification process or to have an alternative to what they are trying to achieve.
I just want to share with you my point of view about the meaning of these certifications and what a potential user can learn from them.

Who gets certified and why?

I know of 2 types of companies who get certified, the ones who do it since they think it is something that will improve their processes and practices, and those who do it since a specific market or competitive condition requires it (i.e. CMMI for the US-DoD, CFR 21 – part 11 for the FDA, ISO to compete in the EU, etc).

As you may imagine, the vast majority of the companies I met fall into the second category.

A thing that surprised me at first was that there is no relation to the size of the Company and it been certified (and why should there be?!). I was always under the impression that only big companies sought certification, but then I learned that many small firms take advantage of this common misperception and get certified in order to appear more robust (and at times larger) than they really are.

How do you get certified?

The process is very straight forward.

All certification bodies (CMMI, ISO, FDA, etc) have a set of publicly available documentation that explains how to comply with their criteria. These documents are basically a (very large) set of checklist with explanations, definitions and even examples about each points that is evaluated during the appraisal process.

A company seeking certification will either get an External Consultant or someone in-house to make a Gap Analysis showing what’s already available in the process and what areas still need to be defined and/or improved. They then create a task team to define and implement the missing processes.
When the new processes are implemented a follow-up analysis or Appraisal Dry Run is performed to make sure that the Company is ready, and if all goes well it requests a formal appraisal to be made.

The appraisal itself is usually done by a group of certified auditors that come to the premises of the Company for a couple of days and perform random checks on the process and specially the documentation that accompanies this process.

They look for a logical and continuous process that is correctly documented and followed in the different groups and products of the company. Based on my experience they look for traceability and usually like to perform wide reviews (take one product and make a complete run from start-to-end) instead of deep ones (take one part of the process and make sure all products have the same docs), but I guess that this can change between auditors and certifications.

At the end of the audit the Company gets an Appraisal Report. It contains results and comments for each part of the process, and it almost always includes feedback with improvement suggestions or requests.

The Company is not required to do PERFECT and thus if it has non-critical issues it will still get the certification, although it will be required to show improvements for these issues in follow-up audits. If the auditors DO FIND things that are critical, they will provide this information and suggest improvements to be done. After their implementation the Company will need to undergo the whole process once again.

Most certifications require the Company to undergo one scheduled audit and a number of un-scheduled audits a year. But this changes between each certification body.

What do we gain from the Certification?

Using my typical sarcasm, the biggest added value of the certification is the Certification itself. The Company will be able to brag about it or show it to a potential customer who will be able to mark-off that specific check-box on their selection criteria.

Taking my sarcasm mask off; a Company working under a defined and thought-off process will be able to waste less time juggling and trying to re-invent answers for common process-related issues.
As an Organization grows and starts having multiple independent groups working in parallel it becomes harder to look for synergies or even simple re-uses of knowledge, experience or code; this can be solved by having a standard process and information systems that everyone knows and uses.

There are obviously more advantages to a certified process; such as repeatability, forecastability, visibility and control; but since they are fairly trivial I won’t review them further here.

What are the problems with Certification?

The main problems are those related to the misunderstanding of the customers and the way many Companies take advantage of this.

The Certification doesn’t mean about the product; it doesn’t say anything about its correctness or even its quality. You can develop a product that doesn’t answer the requirements and is full of bugs, but as long as the documentation is in place and the process is followed your Company may still get the certification stamp. Most times users don’t understand that they still need to validate the product and test it before they buy it.

Additionally, certifications mean a lot of paperwork and red-tape (even if today it is Virtual Paper and Information-System-based Red-Tape). This means that a Company seeking to be CMMI or FDA certified will need to “pay a penalty” in flexibility and its ability to juggle solutions quickly out-the-door.

My personal take is that Certifications are a necessary evil )
In principle they are good and they make us work in a more serious way, but they come with a high overhead that needs to be understood before jumping into it.

If a Company is looking for process improvements they don’t need to get certified, if there is enough management support they will be able to achieve all the added value of the certification with a lot less pain and red-tape by defining a Process Leader in-house.

A couple of QA & Testing Forums around the Web have been discussing the question of how to train a tester, especially during the beginning of his/her career.

This is one of the most important and challenging question any good QA Manager should ask himself; if not for the sake of his Company and his Team, then for that of the tester, who most probably got to this specific Job as a stepping stone into a more “profitable and exciting” career, and with whom we have only a small window of opportunity to show him the real difficulty, dept and challenges hiding under the responsibilities of a QA Engineer.

I will concentrate on the training I believe should be transmitted during the first year of work, and at the end I will also mention a few thoughts about the follow-up processes for later years.

How to train?
I believe the best way to get someone up and running is by mentoring; this is the fastest & most effective way for a rookie engineer to get a good start. Mentoring also allows the person to learn in his own pace, to ask question and to see the more dynamic side of the Testing Work.

I have 3 recommendations around this method:
1. Implement a mentoring rotation, having your rookie engineer working for some time with different testers performing different tasks. This way he will get to see more aspects of the testing work.
2. Make sure the mentors know what they are doing. You don’t want to create the wrong impression by matching a new engineer with someone who either does not know how to communicate his knowledge and/or for some reason cannot invest the required time on his “student”.
3. Even if you implement a mentoring rotation, define a Senior Engineer as the principal mentor (or Big Brother) of the Rookie for the first year in the organization.

One-on-One mentoring is a good first step, and it serves mainly for the introductory part of the capacitation. After 3 to 4 weeks it is good to let the tester start working on real projects by himself (still under the supervision and help of both his mentor and his manager).
He should work on 2 to 3 projects (and around 12 months) “getting his hands dirty” both on the product and the testing before moving forward.

What to train?
Think about it, what makes a good Tester…?
I wish there was a way to teach common sense, but I guess that this is something to filter out during the interview process before the hiring.

Leaving my sarcasm aside, for now, there are at least 3 different aspects I think are essential for every starting engineer:

1. The Application Under Test (AUT)
By this I don’t mean only the system that he will be working on, but all the information needed to do a good testing job.

To name a few, the things I recommend reviewing are:
> Application History & Evolution – what brought the system to be what it is today?
> Company Culture & Values – what brings customers back to our product instead of going to the competition?
> Competitive Landscape – what’s the alternative to our product, where are they stronger, how is the market shifting, and what are we making to be ahead of the curve?
> Installation & User profiles – who uses our application, what characterizes our customers and their Organizations?

2. Testing Methodologies
This is the Strategic Part of the testing process.
How do you plan a testing project, what should be tested and how, what are the considerations when planning and specially when taking risks, etc.

The idea is not to have your rookie engineer plan his own testing projects up front, but he should understand the reasons for the operations he is performing.
Assuming the person you hired has enough common sense he will work better if he understands how his individual efforts fit into the complete scope of things.

3. Testing Tools (and I am not talking about QuickTest or Robot!)
You want your engineer to start working as soon as possible, so make sure he knows what he is expected to do and how. There are 2 types of tools, technological & methodological, and you should make sure he feels comfortable with both.

It is not logical or practical to try and teach all the tools at once, but as part of his mentoring rotation your engineer should get to know at least 3 to 4 of each relevant type.

I recommend also making sure that even if someone was hired to do a very specific type of testing (let’s say stress testing using LoadRunner for example), during his initial weeks in the Company he will get to know the other types of testing done, since this will give him the perspective needed later in his project.

Follow-up training and certifications
Just for the sake of transparency I am ISTQB CTFL and a member of the ITCB Advisory Board, but my views on formal training and certifications where made up some years before I even though of getting certified.

I think formal training for a Rookie is almost a complete waste of time. Without enough experience it is very difficult to understand why we need all the tools, what are the advantages and disadvantages of each, and most importantly when should we use one instead of the other.
This view goes both for testers who want to start their career by working with an automation (functional, API or load) tool as well as those going for certifications before having at least a solid year of manual testing experience.

Training (and here I include certifications, hoping no one does the exams just to have a diploma hanging on their walls) are an essential part of a testing career path. As an engineer starts working, he will understand what are his strong and weak points and that way he should decide what path does he want his career to take.

I will stop this point here, and probably re-take it on a future post…

Bottom Line
If you want to have good testers you have 2 alternatives, recruit people already with experience or recruit people with potential and invest on them in order to help them grow into good Test Engineers.

Training is a project on itself; it needs to be planned, executed, measured, reviewed and improved.

The worst mistake you can make when recruiting a new engineer is to let him sit on his desk and expect him to start testing without any guidance or training. It is the same as buying expensive cooking materials and placing them in the table to be eaten as-is; materials are a critical part of cooking, but not less important is how you cook and prepare them.

Here I come again, another customer who asks for QA Process Consulting.

Again, like many start-ups, they’ve been using Excel to manage their issues but feel they outgrew it and now need a “real tool”.

Again, 10 minutes into the tool evaluation meeting someone says those frightening words I’ve heard at least 5 times before in my testing life “How about Bugzilla, it’s free and even NASA use’s it so it should be good enough for us!”At that point in the meeting I try to explain about the pros & cons of Bugzilla but I guess the price factor has temporarily dazzled my stakeholders.

I finalize my inputs to the discussion with my old grumpy saying: “There’s no such thing as a free lunch, but if that’s the way you want it I’m willing to help you give it a try”

Starting by the end (or at least the current state after 8 hours of work):

- The installation on top of Win 2003 with IIS 6 took us close to 6 hours to complete (4 of them trying to make it work with IIS, before switching to Apache); and we are still having problems sending mails.

- Once the system was up and we started configuring it they understood what I previously meant by limited customization capability. We have not showed it to their internal users but I am willing to bet the responses will be something like “What was wrong with the Excel? It was a lot more user friendly that the new system…”

- We also need to start thinking how to import the data they already have – I hope the XML import mechanism really works this time.

- Last but not least, once they start working we will need to understand how to generate graphs & reports to let them do something with their data. Anyone with experience and/or ideas around this last point is more than invited to add their comments (please…?)OK, enough with the melodrama and let’s put the things in proportion.
We are talking about a robust system that has been working for many years, thousands of projects have used it, and hundreds of good developers have and continue contributing to it.

So how can it be that bad?

Actually Bugzilla isn’t that bad, I am even sure that on certain days (depending on the moon) I may go as far as to say that it is a pretty good product; but I believe Bugzilla is good ONLY for a very specific type of organizations and teams: Teams of developers with a development oriented way of thinking.

I realized this when reading the first chapter of whysoftwaresucks and David Platt’s explanations of what happens when Good Developers take charge of the User Interface (and not only the GUI).

These guys don’t really see the need to make something that meets their End Users requirements, if it is logical for them then it should be logical for everyone. I guess you understand that this is where they are wrong, but I won’t go into more details since you can always read the book.When you work with a system that was written for free you should not expect it to meet all your needs, you should not expect it to have all the functionality you require, and you should not expect it to make your life easy; after all these guys did it on their nights and weekends so why should you expect all this?

You will need to take into account that (this is a partial list):

1. It will require you to install it and maintain it; and since the people who developed it are highly technical they will not think about including wizards and graphical interfaces to make your life easier.

2. The customizations will be minimal. If it comes together with the code and you can modify it as you like, why do you need for that?

3. The issue workflow will be rigid and will only cover main path scenarios. On top of the previous reason there is always the answer that overcomplicating the workflow only makes processes harder; which is true for many cases, but 90% of organizations have REAL NEEDS to modify their workflow and to do it by twitching the code is not a possibility for them.
4. The GUI will not be easy to understand by your non-technical End Users.

If the application was made by developers and they are good at writing code (instead of their Graph Designer counterparts that are good at designing screens) don’t expect it to be friendly.

5. Reports and graphs are things you can go to the database and fetch for yourself using simple queries, so why bother with a reporting engine?

6. Now, about support… It doesn’t really matter that there is a large community of users, sometimes you need someone to work with you and troubleshoot the problem in your system, this is usually not the case for free software.

There are more points like the ones above, they all point to the conclusion that if something comes for free it means that there are many compromises you will need to make.My bottom line is that there is no such thing as a Free Lunch – there is a price to pay and people tend to overlook it.

I think this is also the reason most companies I helped install Bugzilla called me after 3 to 6 months in order to migrate them to a more serious tool.

This might simply mean that sometimes improvements need to be done using baby steps…?