State of Software Testing 2013- Webinar transcript

Hosted by: Lalitkumar Bhamare, Founder of “Tea-Time with Testers” online testing magazine
Facilitated by: Joel Montvelisky, Head Solution Architect at PractiTest

(blog: – highlighted in brown

Star Panelists:

Jerry Weinberg

Jerry Weinberg (highlighted in green)

fiona charles

Fiona Charles (highlighted in purple)

Introduction (by Lalit)

Lalit: Welcome to the State of Software Testing Webinar, brought to you by “Tea-Time with Testers”, in association with Practitest, with star panelists Jerry Weinberg and Fiona Charles.

  • Jerry Weinberg is a prize-winning author of 80 books, international consultant, speaker, teacher, expert, and coach in all phases of software engineering. He is also the founder of the AYE Conference.
  • Fiona Charles is a software test consultant, teacher, and writer, and she assesses, coaches, renovates, and rescues testing of any scale. You can follow her on twitter @FionaCharles


Main topics:
– (Perception of) Testing as a Profession
– Learning and Professional Development
– Testers and the Extra Activities they perform
– Software Development Models
– Is Software Testing a non-promising Career?

Full Webinar Recording:


Background Information (by Joel)

The State of Testing Survey is a shared initiative by TeaTime with Testers together with the QA Intelligence blog. The driver behind this project was to establish a source of objective data and information (at least as much as possible) that will provide an insight into what is going on in the testing world today.

The survey itself ran for a number of weeks during December 2013.  We gathered responses from hundreds of testers from many countries and organizations, and the results were published as the “State of Testing Report” in February, 2014.

Our aim is to run this survey again once a year, allowing us to gain insights into changes in the way the testing world behaves, helping us to understand some important trends that are shifting the practical reality of our profession.

The idea of today’s webinar is to perform a review of the State of Testing Survey, and together with the help of Jerry and Fiona, analyze some of the main results gathered from this first edition of the survey.

(Perception of) Testing as a Profession

Joel: We want to start with the topic of Testing as a Stepping Stone into Development and IT.

In the past, at least, many of us thought that people and especially young engineers saw testing as a stepping stone into other fields in the Industry.

Still, from the results of the Survey we saw that most (responding) testers have been working in the field for 6 or more years, meaning that this theory of working for a short period of time in testing seems to be wrong.

Another thing we were able to see from the Survey’s results is that a lot of people are not switching working places at a quick rate, but rather staying in one, two or three organizations overall.

What does this mean to this previous perception of testing today?


years testing              Companies Worked At



First of all, we need to be careful on how we interpret this data, for example we know this is not a random sample of all testers, testers working for less than a year might not be looking at the places where we advertised this survey.

The important thing is that we have a lot of people with more than 3 years in testing. On the other hand 3 years is not really a lot of time. Personally, I have been in the business for over 50 years, and when I was 3 years into it, I was just getting started.

Having said that, some years spent in software testing is the best possible preparation for moving to other jobs, if you want to. There is absolutely nothing wrong with staying in software testing your whole career, if you want to, except maybe the positions and rewards are not so available.

This represents a fairly healthy situation right now – people get some job security, they can go to work for a company, they’re not just hired to test one project and then gotten rid of.

They can come, stay in a job for a while and learn a whole lot, and prepare themselves for whatever else they want to do, because testing is the number one place to learn about the whole software development business.



I’m a bit dubious about the perception of testing as a profession, rather than perhaps the perception of testing as a career.

The term profession has a quite specific meaning, which is not necessarily the same thing as a career choice.

I’m kind of interested in the perception that in the past testing was considered by some as a stepping stone. Yes, that was true, for some, but testing has been a pretty reasonable career choice for quite a while. I’m not sure whether we are seeing a change or not.

I’m also not sure how you correlate the “years testing” graph with the “companies worked as a tester” graph. Is there any correlation….?


Joel:  Well, we actually did a high level review of this point and although we also thought there would be a correlation between the answer groups, we saw, to our surprise, that there actually wasn’t.


Fiona: If you look at how testers are employed there are at least three kinds of options:

1. Testers who are in-house, who work on a variety of projects, as they come up, or in fact work in sort of regular operations.

2. Testers who work for consulting companies, where they get experience across a broad range of clients.

3. Testers who are independent.

It’s hard to understand what the data is actually telling us here.


Joel: One of the things we understood is that we are only starting to understand what we need to ask in the surveys in years to come.


Jerry: One way to look at the survey is that it’s giving information about what information we should be getting. It’s a preparation for next year’s survey and the one after that.



Learning and Professional Development

Joel: Most testers use self-learning or on-the-job training in order to learn. This is not something that changes as people become more professional.

We are trying to understand how this fact compares with reality of the peers of software testers. In our case, we work with programmers, analysts or people who actually have computer science degrees.

How does this affect testers who (don’t have formal training in testing) and actually come from a self e-learning perspective? Is this something that benefits testers or is it a disadvantage for us? Also, what can we do about it, if anything at all?


 How dis you learn to test           Testing Education



When I started in IT as a technical writer in 1978, nearly all of the programmers I knew were people who came from a huge variety of other disciplines and to a large extent were almost self taught.

The computer science programs were relatively young then, and it’s only in the last 20 years that we’re seeing that the majority of programmers and analysts come out of computer science degrees.

Having worked on software projects over that time consistently, I have seen an increasing narrowing in how programmers and analysts approach the development of software. I think there are some real benefits in the increased technical training, but there are some downsides in that I don’t see some of the same level of empathy with customers from computer science graduates that I saw when they came from a variety of backgrounds. Many technical people develop that empathy over time.

I wonder if the difference in the education of testers and programmers doesn’t in fact give us (testers) an edge, in terms of our ability to relate to the people we’re developing software for. It’s a very important part of our job to be able to have that empathy and to be able to represent the interests of the stakeholders that we’re developing the software for.

I don’t see the way we learn to test as necessarily having a major downside at all. I think that there’s a lot of value in the kinds of ways that we go about it. Obviously, on the job training can have a tremendously variable quality. It can consist of being given a template and being told to fill it out, or it can consist of being given challenging projects and being mentored through those projects. On the whole, I’m not sure we’re in a bad position.


Jerry: I would have thought that on the job training would be 100% because I can’t imagine doing a job as a tester and not learning something new everyday. Who are these 25% of people who are working as testers and are not learning anything about testing from doing it? It’s inconceivable to me.

Maybe in the follow-up surveys, we might ask more specific things, like how many of you have read a book on testing in the past year. The usefulness here is whether you’ve done a certification or taken a formal course. I’m not worried about those numbers being too small – I don’t have a lot of confidence in formal training in testing and in the certifications that exist. I think any reasonably bright person in a few months of working as a tester will pick up pretty much everything that these courses can cover.

I can see people learning new specialized techniques or learning new tools as being useful. There are so many different tools, that if you take a course in a university about tools, they’re most likely not going to be the tools you’re using on the job. The learning that you want is to learn the diversity of tools, and learn things about how to tell what’s a good tool or not and which things to ignore in a tool. We’d like to know where specifically people are getting this formal training and what it consists of. Right now, it’s not a very big part of what people are doing, and as far as I’m concerned, that’s ok. I would like to see on the job training up to 100%. I’d like to find out whether people are doing things like a reading group at work, for example.


Joel: How should a tester today continue growing? What are the recommended tools they can use to continue improving their testing techniques?


Fiona: I am looking at the variety of things in this box (refer to diagram above) and I think to myself, do them all! I have trouble imaging a tester who is not learning constantly. There are so many things that contribute to your education.

The field is changing, there are new ideas, the technology is changing. Suddenly, people need to learn about mobile, for example.

You need to keep up and understand that there are ranges of thinking in testing and there are also a number of testing schools, and you need to understand what each group is doing that can help you become a better tester.


Jerry: I can’t imagine someone who is working as a tester and doesn’t subscribe to at least one testing magazine, especially since at least one of them is absolutely free. There is simply no excuse.

The same way with online communities and forums – they are free and there is no reason to not join.  If you belong to one that is not useful, you leave it and join another one.

The kind of thing that I recommend to testers, as far as education is concerned, is not specialized courses for testers, but instead courses in communicating better with people, better problem solving, and other general skills like that rather than particular testing skills, because you’re doing that on the job anyway. Most testers that I know are insufficiently trained in communication skills, they don’t know how to write even a good bug report, they don’t know how to talk about the things that are difficult to talk about because people generally don’t want to hear that they have bugs in their software.

I think that Weekend Testing is a very interesting category, but I am not sure everybody is aware of what this is.


Fiona: One of the things that I recommend to people is to seek new projects that will teach them new skills and give them the opportunity to learn new products and testing skills. Look for people you want to work with and learn from them.


Joel: Another skill that testers need to learn or sharpen is their technical skills.  It can be as easy as sitting with a developer you like and asking him what is he doing, and trying to understand more about the technical aspects of his work and how can that knowledge help you be a better tester.


Jerry: Along that line, the biggest thing you can do is to choose your job. If you are not learning from your job, then get a different job.  You don’t want to be in a job that doesn’t contribute to your learning. If you are not learning every day you are in the wrong place.



Testers and the Extra Activities they perform

Joel: This takes us to the next slide: What other kinds of things do you do as part of your job? We see that people are managing the testing and development environments, working on customer support, or working on requirements.

But, why are testers doing extra activities? Is it that we have a lot of spare time or are these things meaningful to what we do?  How can these activities either help or harm you as a tester?


Roles of Testers


Fiona: I don’t see how anything related to software development can harm you as a tester. I think acquiring coding skills is only going to help you understand more about how software is developed in the first place, and perhaps help you get inside it.

Gathering requirements is a wonderful place for a tester to be to better understand it. How many testers do you hear talking about how awful their requirements are?  If your requirements are not good enough, go and be part of the gathering and analysis process.

Customer support is also a great way of understanding who is using the software and how.

Where I get frustrated is where testers are part of meetings that have nothing to do with their projects or when they have a lot of paperwork that is not related to the development of the software. Any activity that is linked to the development of the software can provide an excellent opportunity to learn and develop yourself.



If you work in an organization where testers are not involved in the requirements gathering, you should be working to get that fixed in your organization. If you can’t (fix that) then that is a good sign you should be working somewhere else.

Most of the requirements that I see are not testable.  Words like fast, portable, etc. What does that mean for you as a tester?

What you are doing when you are testing – you are trying to find places where software is suitable or not suitable for use. In order to do that, you have to know about how it’s used. Many of the things we test are used in very special environments and we know nothing about them.

If you’re involved in requirements gathering, that might give you some sense of how the software is being used.  If you participate in the deployment you can also get a lot of information from users, and you cannot learn that just by studying the software. Customer support is such a great place to learn that I advise testers to spend one or two days every month working on the phone with customers. How else would you know what’s really important to people?

You can’t test everything – you have to make choices as to where to put your effort. You should put your effort where it’s important to your customers, and you have to know how they use your product. You learn this either out in the field, installing the software, or on the phone, hearing what people are having trouble with.


In summary – all these extra activities are parts of the tester job description:

– Customer support

– Participating in the installation or upgrading of software

– Being out there with the customers and users

– Participating in requirements gathering and reviewing

– Managing your testing environment

If this is not part of your job description, start working to get it as part of your job description.



Software Development Models

Joel: What software development models are people using?

78% declared agile or agile-like.  We also see people who say that they are working based on a number of methodologies at the same time, and we even see as much as 12% of responders who say that they are not using any structured development methodology.

Focusing momentarily only on agile, is Agile our holy grail model?

And then, if you look at those who are not following any model, should they be looking for another place to work?


Agile Adoption trends


Jerry: Right now the magic word is Agile, and so if you ask them they will tell you they work Agile.  But if you look at what people are doing, they are testing.

Over the years, I’ve seen that whatever the latest fad is, people will claim to be doing it or something just like it.

The real question to ask testers is:  “Do you know what your developers do?”

1. Do they do the same thing consistently?

2. What is it that they do?

Rather than pay attention to what they say they are doing, I’d ask that you find out what they are really doing.

If you know what they actually do, then you will be able to understand the things that they are not doing that may get your testing to be more effective.  Remember that process description is not the same as process, and you need to understand what it is that they are doing and what does that mean to them.


Fiona: I am not sure it really matters how organizations do their work, but it matters if we are being let to do our jobs in a good and valuable way.

I don’t believe in holy grails. I’ve seen one method after another sweep through software development, and every one of them contributes something. What we often find over time is that the good things become part of the culture.

For example, I hope what we will see coming out of agile, when the next method sweeps through, is smaller increments of software development and more frequent iterations, rather than great big bangs. I think it has been demonstrated that we get better software that way.

There are agile teams out there that say they don’t want testers, that they don’t see a need for professional testers, and maybe they don’t need them.  So we need to look at this and understand that maybe not every team needs to have testers, for example freeware of software with low risks.  So the bottom line is that we need to understand that our job is to help produce excellent software (even if this doesn’t answer your question directly)


Jerry: To quote Abraham Lincoln’s favorite riddle – “If you call a tail a leg, how many legs does a dog have?” The answer is four, because calling it a leg doesn’t make it a leg.” Calling it agile doesn’t make it agile! In the same way that calling tested doesn’t make it tested.

Name magic is not a substitute to doing your work.



Is Software Testing a non-promising Career?

Joel: We asked people if they are concerned with their jobs as testers and 42% said they are not concerned, while 58% said they were somewhat or very concerned.

Is this something to be worried about or to really be concerned?

What does it mean to us as testers?

Are you concerned about your job?

Concerned About your job

Jerry: I’m more concerned about the 42% that are not concerned. There are too many people building software who just think of testing as a nuisance, as just something they need to avoid or work around in some way. This is something to be concerned about.

If your team does not value what you’re doing, you’re not going to get rewarded for what you do and you won’t have a lot of job security.

Part of your job is to help people understand how important this job is, what you do, by having examples to show them what happens when you test well and what happens when you don’t.

We need to learn how to do this and then be less concerned, but I feel that right now people don’t realize they should be concerned about this.


Joel: Following up on Jerry’s comment, what is testing’s added value?


Fiona: Testing done well provides you with insurance. Some places need more insurance and some need less, it depends on the risk of your product and environment.

Testing does’t guarantee that you won’t release without bugs, but it helps to minimize that risk.

Not all testers are good testers, and the best way to persuade people of the added value of testing is to do it excellently!


Jerry: Testers can learn about the insurance business. If you understand the logic of insurance, you will understand the testing business better.   You need to understand that risks are the logic behind insurance.  If you understand what needs to be insured then you will understand what needs to be tested, and also on what level.

Part of our job as testers is to see what deserves the level of testing we are offering and sell that knowledge to the other people in the organization.


Joel: Thank you very much to Jerry and Fiona!… and to Lalit for putting this webinar together!