PractiTest QA Blog

Many testers feel frustrated when they are asked what is their Job, and specially about what do they contribute to the overall development project.

So, think about it… What are you expected to contribute in the scheme of your Product Development Project?

Are you supposed to catch all bugs before releasing the product to the world?
I think Not. It is exuberantly expensive to reach this level of bug-free-environment, and 99% of the projects don’t need it anyway.

Do you need to run as many tests as possible?
The answer again is NO! If you could get the same information (or confidence level) about the product without running a single test you would still be doing your Job – maybe even in a better way!

The real added value we provide as testers is something called Testing Intelligence™: Timely, accurate and actionable information about the status of your product and project.

In simpler terms (as I defined in a past article), we need to provide correct and timely visibility into the product and process, that will help the Organization make the correct tactical and strategic decisions.

I call it Testing Intelligence since the process and its outputs have many similarities with what is known as Military Intelligence:

1. It is based on the principles of gathering information from multiple sources, and mixing it to create a model representing the current situation and status.

For example, in order to give a complete estimation of your project you will use the data from:
- Test results showing pass vs. fail numbers and percentages
- Total number of bugs opened, fixed and closed
- Load benchmarks for your application under different levels of load
- Test execution and bug resolution trends to indicate the progress of the project and the way it is coming along based to the original plan
and
- Historical data for similar projects, to serve as a base for comparison

You will then take all this isolated data points, you will create direct and indirect correlations, and then transform it into structured information (e.g. a Report) that provides a better image of the product’s status and of the project’s chances of meetings its schedules and targets.

2. The results of your analysis cannot be considered an exact science – by you or your stakeholders.

Since you are constantly making assumptions based on incomplete data, it is normal to expect to be wrong part of the time.
Each statement provided should come with an associated level of risk (and in some cases an explanation of the source of such a risk) that will allow your stakeholders to make more balanced and informed decisions.

3. The way you understand the situation will change as you become aware and analyze new data and information.
You should not be afraid or ashamed to modify your conclusions and provide an updated model that describes the situation better, you need to explain why this happened and how it affects your previous estimations.

4. The way you present the information needs to be defined by the needs and likes of your stakeh0lders.

If they expect to see only numbers and a short conclusions don’t go around publishing long pages full with explanations. If on the other hand they like reaching their own conclusions, you should provide them a report with your data, followed by your assumptions, and ending with your conclusions; then you should work with them and let them agree or disagree with your results.

In any case the important thing is to give them what they want, but to make sure you don’t paint a picture prettier or uglier based on their personal believes or interests.

Working with incomplete information does not give you the right or an excuse to make a sloppy job. It is actually the reason why your professionalism, self-criticism, and 360˚ understanding of the situation is even more important for your work and the well-performance of your Company.

Be proud to be a Tester, your job (if done correctly) is more complex and demanding than most of the other members of your team.

Hello all!
Usually this blog focuses on QA methodologies, test processes, testing tools, platforms and QA best practices; but from time to time, we will also talk about related topics such as: technology, IT, SaaS (Software as a Service or On Demand solutions) and more.

Many people asked me what are the key things to keep in mind when developing a SaaS application. These are the main points that guide us when we build the PractiTest platform:

• Scale, Scale, and how to scale - that’s one of the main issues to think about BEFORE starting with any piece of code
  How can I make it scale?
  How much will it cost (per additional customer / project / server)?
  Will it add additional maintenance tasks?
  How can I make my servers Highly Available?
  What will happen if the platform will require more than one DB?I know that Agile development is all about how to start the development before you make too many design documents; and not starting and fixing things that aren’t broken yet (and maybe will never be!).

  So maybe I’m a little old-fashioned, but in my opinion to start and code before taking into account any ’scale thinking’ is irresponsible. I think that regarding scaling we chose the middle way. Some examples:

  • We did not decide before writing the code what would be the indexes in the DB (for example), but when we create the tables, we put some relevant indexes, even when our DB is already working very fast.
  • We made sure that scaling can be achieved by adding relatively low cost machines.
  • We use opensource technologies to make sure scaling by adding machines will not require many $$$ for licensing.
  • We made our code ready for S3 (for file storage) if required, so we will not need to spend many bucks on storage solutions.
• Security – unlike traditional SW, where the DB is located on client’s premises, and all the communication is usually only inside the LAN of the organization, in the case of SaaS systems everything is over the web.
  Security is one of the most critical items, since you are storing and transmitting your clients’ data. Your code needs to forbid ANYONE IN ANY CASE to get or update data unless he’s identified as part of that specific account. The key here is strict standardization within your code.
  We built PractiTest in a way that even WE can’t manipulate or see any client’s data if we don’t have the required permissions.
  Before building any web application, I think anyone should learn about (and how to avoid) things like SQL Injection and Cross Site Scripting.In addition, security also means respecting the users’ privacy and sensitive data:
  • Hashing users’ password (MD5), before saving it to your DB.
  • Filter sensitive information from the default logs (credit-card info, password).
    Build it in a way that even you (or the system admin) will never be able to find users’ sensitive data.
  • Get an expert to try and manipulate your data.Special Tip for testers:

    try to find a user password on the DB or logs, or try to manipulate/get data that you’re not allowed to access, using http post or get. If you succeed it means there are issues with the product. These issues are showstoppers for every SaaS products, they are also important to fix in any SW product.

• Fast upgrades – SW upgrades shall not take more than few minutes. This should include DB & files backup, latest code version backup (in addition to version control), modifications to the DB schema (if required), upgrade of the code, and sanity testing.
  All the above should be automated. As a rule of thumb, the update should last no more than 10 minutes. If there is a failure, make sure you can rollback within 1 minute. Fast rollback is the main reason we backup DB, files, and code version, before we start the upgrade.
  As a SaaS solution you are expected to quickly solve every problem that may occur; also adding modules is something that you should be able to do easily.
  For example, in PractiTest, we usually do bi-weekly updates since we are adding features all the time.
• Make It Work Fast! The last thing your users (and ours) want is to wait (while you load your application). Actually some customers choose us mainly due to the fact that PractiTest works much faster than the products they used to run in-house.
  How could that be if they run it on their local network!? These are the things we keep in mind:
  • Small javascript files, small download pages.
  • We use only one JS framework. More than one can make the browser ‘become heavy’.
  • We make sure we have enough network capacity to support our clients.
  • We constantly check DB queries on production. Making sure all run fast.Our platform already works very fast for our clients, and this is even without (yet) things like:
  • Javascript and CSS compression (such as YUI Compressor or Packer)
  • Browser image cachingFor additional techniques you can look here: http://developer.yahoo.com/performance/rules.html
• No Installs for the clients – When you are developing a product that is installed on the customer’s premises, you can sometimes get away supporting only IE with an ActiveX based product. But as a SaaS your platform needs to support all main browser without instaling ANYTHING.
• Adjustments as needed – this is only one of the SaaS advantages – if there is a problem, fix it on the fly.

As I mentioned before, this is obviously a partial list of our key points. I mainly wrote what to do and not so much how to do it properly. If you want to know more about how to do it (or how we do it in PractiTest) drop me an email or post a comment.

Yaniv

I apologize up-front for the one-sided localization of this post )
But I am sure all our Israeli Readers will enjoy the newest addition to the PractiTest family – http://heblog.practitest.com (our Hebrew language Blog)

Kudos to Yaniv for starting this endeavor.

Yaniv promised that this blog will touch subjects around Software Development, SaaS, technology, and all sorts of other stuff directly or indirectly related to PractiTest.

GO AND CHECK IT OUT!!!

(I wonder what should we do next, Spanish or Russian…?)

I’m amazed that some people still think of their Bug Databases as “Garbage In – Garbage Out” systems where to STORE bugs as they become available.

When I review the subject objectively, I think these people are both Right and Wrong at the same time.

Since I am a positive person I will start by explaining why they are RIGHT

If you treat your bugs like garbage then that is all they’ll ever be.

If you don’t take the time to define a defect life-cycle that fits your specific needs, if you don’t think ahead about the information you want to capture from each issue when it is reported and handled, and if you don’t do something with the data that “sits waiting” at your fingertips; then don’t expect it to be worth anything at all.

As my father thought me: “nothing good in life is free, if you want to get value you need to work for it”.

Now, let’s review why these people are also WRONG

A well defined and managed Bug Database can be one of the most important assets of a Development Organization.

1. If you use it to manage not only bugs but tasks, enhancements, and all other action items of your team (someone said Agile?); it ensures that everyone is working on the most important things for the project all the time.

2. By keeping track of the rates for bug detection and resolution you are able to understand whether your project is on-track or slipping out of control for its expected release dates.

3. If you think ahead of time about the information you will want to analyze during your project retrospective you’ll be able to collect the data in real-time.

4. If you keep track of your escaping defects and perform root cause analyzes of the reasons they were released (remember to stay clear of witch hunts!), they become an Internal Engine pushing forward your On-Going Improvement Efforts.

5. When you use your Defect Database in conjunction with your CRM (customer relationship management) system – and most importantly when you bring in the issues and enhancements reported from the field – you are able to leverage this information in order to plan, design, develop and deliver the features that will help you to advance your product in the eyes of your customers.

And the list goes on and on… so I will stop in here.

I think it is time we realize that we (testers) are in the business of Process and Product Intelligence. Our bugs are an essential part of the data we use in order to reach concussions and do our jobs better.

Treat your bugs like garbage and that’s all they will ever be, it’s up to you.