How we test our SaaS QA Platform

For those who are not up to date with the latest buzzwords SaaS stands for Software as a Service; it means that the software platform is hosted and managed by the vendor and users simply open their browsers and connect to the application from anywhere in the World.

Part of my daily activities include being in charge of the QA and Testing for PractiTest, a SaaS QA Management System used by QA and Testing Teams to manage their Test, Bugs and Process.

One of our customers asked me how is testing a SaaS platform different from testing any other project or product?
In principle I don’t think it is different, but it requires us to merge various types of tests that up to now I’ve only used on isolated projects.

To give a better explanation, here is a very quick list of the types of tests we constantly run on PractiTest:

Functional Scenarios – As with any other application we make sure that the functionality works as expected.
For this we use the regular testing techniques such as:
– Manual test scripts
– Exploratory charters
– Checklists
– End-to-end scenarios
etc
Apart from manual testing we are also starting to *play* with selenium in order to create an automated regressions testing suit.

Multi-platform support – We use a combination of physical and virtual machines to make sure PractiTest users can work on Windows, Mac and Linux; an also on the IE, Firefox, Chrome and Safari Browsers

Load and Stress – PractiTest needs to handle large amounts of users and we don’t have the luxury of re-booting or going down once in a while.

Remote Access and Usage – Simply put, we make sure that all users regardless if they come from the US, Holland, India, Argentina, or Australia can work with the system with good response times.
There are many emulators (such as Shunra) that can help test this in your lab, but there’s nothing like the real thing, and for this we have a number of local collaborators that are willing to perform a set of tests and report their results from their locations.

Security – We make sure no one can harm (consciously or by mistake) their system or the platform. For this we test that things like cross-site scripting and other security loop-holes are blocked.

Live updates and deployments – Here is something we seldom think about in regular applications: How do you deploy the system while it is still running, and if needed how do you minimize down-time for the users?
This is something that is developed and handled by our IT department, but since the delivery and update of the product is part of the overall user experience we test it constantly.

Disaster recovery & rollback procedures – Another testing task coming to us from the IT side of the house.
Here we run 2 main scenarios:
1. System down that needs to be brought up quickly (with the same machines or with new ones that require installation and configuration)
and
2. Rollbacks to the last known stable version, including data.

I18N – Since our platform is used by people around the world we make sure that we support the use International Characters.

There are additional tests and activities we are constantly adding to our testing suites; but the main principle is that we’ve learned to see our AUT (application under test) as the complete end-to-end user experience and not only the functional application where testers (our users!) do their work.

We understood that we don’t test only to make sure the functionality works. Our testing objective is that users get the best possible APPLICATION AND SERVICE from our company.

About PractiTest

Practitest is an end-to-end test management tool, that gives you control of the entire testing process - from manual testing to automated testing and CI.

Designed for testers by testers, PractiTest can be customized to your team's ever-changing needs.

With fast professional and methodological support, you can make the most of your time and release products quickly and successfully to meet your user’s needs.

4 Responses to How we test our SaaS QA Platform

  1. karthik May 15, 2009 at 1:13 pm #

    excellent info…btw i have been looking up distributed testing – eg:http://tetworks.opengroup.org/Wpapers/distributed_whitepaper.htm
    This may not be relevant for your product, but do you think this is something that might be useful for testing saas products?

  2. Joel Montvelisky May 15, 2009 at 1:39 pm #

    There is confusion in the different meanings given to distributed testing.

    Personally I’ve heard the therm around at least 3 different things:
    1. Systems with mirrors around the world, that require us to make sure the synchronization and interaction between all the servers works correctly.
    2. Systems composed of multiple interacting applications or parts, and here it is more around the integration testing area.
    and
    3. Systems where the operations are distributed. It can be live and shared operation such as Instant Messaging Systems where you need to check that the 2 client (and maybe multiple distributed server) systems interact. Or it can be a big ERP system where we need to make sure the flow is carried and works in the serial workflow defined.

    I believe that each product, regardless if it is SaaS or not, needs to check if it has any of the above needs and plan tests accordingly. And obviously SaaS system may be more inclined in having systems with all of the above 3 needs.

    In any case, these are definitely part of the analysis we perform when we start planning any testing operation in PractiTest.

  3. vidhya February 2, 2010 at 1:15 pm #

    Nice piece of information 🙂

  4. hary March 2, 2013 at 10:33 pm #

    Thank you for good info

Leave a Reply

shares