There’s no such thing as a Free Lunch – my latest Bugzilla war-story

Here I come again, another customer who asks for QA Process Consulting.

Again, like many start-ups, they’ve been using Excel to manage their issues but feel they outgrew it and now need a “real tool”.

Again, 10 minutes into the tool evaluation meeting someone says those frightening words I’ve heard at least 5 times before in my testing life “How about Bugzilla, it’s free and even NASA use’s it so it should be good enough for us!”At that point in the meeting I try to explain about the pros & cons of Bugzilla but I guess the price factor has temporarily dazzled my stakeholders.

I finalize my inputs to the discussion with my old grumpy saying: “There’s no such thing as a free lunch, but if that’s the way you want it I’m willing to help you give it a try”

Starting by the end (or at least the current state after 8 hours of work):

– The installation on top of Win 2003 with IIS 6 took us close to 6 hours to complete (4 of them trying to make it work with IIS, before switching to Apache); and we are still having problems sending mails.

– Once the system was up and we started configuring it they understood what I previously meant by limited customization capability. We have not showed it to their internal users but I am willing to bet the responses will be something like “What was wrong with the Excel? It was a lot more user friendly that the new system…”

– We also need to start thinking how to import the data they already have – I hope the XML import mechanism really works this time.

– Last but not least, once they start working we will need to understand how to generate graphs & reports to let them do something with their data. Anyone with experience and/or ideas around this last point is more than invited to add their comments (please…?)OK, enough with the melodrama and let’s put the things in proportion.
We are talking about a robust system that has been working for many years, thousands of projects have used it, and hundreds of good developers have and continue contributing to it.

So how can it be that bad?

Actually Bugzilla isn’t that bad, I am even sure that on certain days (depending on the moon) I may go as far as to say that it is a pretty good product; but I believe Bugzilla is good ONLY for a very specific type of organizations and teams: Teams of developers with a development oriented way of thinking.

I realized this when reading the first chapter of whysoftwaresucks and David Platt’s explanations of what happens when Good Developers take charge of the User Interface (and not only the GUI).

These guys don’t really see the need to make something that meets their End Users requirements, if it is logical for them then it should be logical for everyone. I guess you understand that this is where they are wrong, but I won’t go into more details since you can always read the book.When you work with a system that was written for free you should not expect it to meet all your needs, you should not expect it to have all the functionality you require, and you should not expect it to make your life easy; after all these guys did it on their nights and weekends so why should you expect all this?

You will need to take into account that (this is a partial list):

1. It will require you to install it and maintain it; and since the people who developed it are highly technical they will not think about including wizards and graphical interfaces to make your life easier.

2. The customizations will be minimal. If it comes together with the code and you can modify it as you like, why do you need for that?

3. The issue workflow will be rigid and will only cover main path scenarios. On top of the previous reason there is always the answer that overcomplicating the workflow only makes processes harder; which is true for many cases, but 90% of organizations have REAL NEEDS to modify their workflow and to do it by twitching the code is not a possibility for them.
4. The GUI will not be easy to understand by your non-technical End Users.

If the application was made by developers and they are good at writing code (instead of their Graph Designer counterparts that are good at designing screens) don’t expect it to be friendly.

5. Reports and graphs are things you can go to the database and fetch for yourself using simple queries, so why bother with a reporting engine?

6. Now, about support… It doesn’t really matter that there is a large community of users, sometimes you need someone to work with you and troubleshoot the problem in your system, this is usually not the case for free software.

There are more points like the ones above, they all point to the conclusion that if something comes for free it means that there are many compromises you will need to make.My bottom line is that there is no such thing as a Free Lunch – there is a price to pay and people tend to overlook it.

I think this is also the reason most companies I helped install Bugzilla called me after 3 to 6 months in order to migrate them to a more serious tool.

This might simply mean that sometimes improvements need to be done using baby steps…?

About PractiTest

Practitest is an end-to-end test management tool, that gives you control of the entire testing process - from manual testing to automated testing and CI.

Designed for testers by testers, PractiTest can be customized to your team's ever-changing needs.

With fast professional and methodological support, you can make the most of your time and release products quickly and successfully to meet your user’s needs.

2 Responses to There’s no such thing as a Free Lunch – my latest Bugzilla war-story

  1. Philk July 3, 2008 at 10:27 am #

    We used Mantis at my last place, it worked well but only because I was an ex-programmer and could easily hack about with the php behind it

    but my main reason for commenting was to point out that your link to whysoftwaresucks actually goes to softwaresocks which gave me a chuckle

  2. Joel Montvelisky July 3, 2008 at 10:41 am #

    Thanks Phil, nothing like a good user to find bugs :o)

    Cheers (& thanks!)


Leave a Reply