SaaS Testing Webinar

This week I was invited by SoftTest Ireland (& more specifically by Anne-Marie Charrett) to present a webinar on how we test PractiTest as a SaaS Application.

I hope the people who attended the webinar enjoyed it. I certainly did, specially after we started the presentation and my level of anxiety went down a little 🙂

I won’t go into the details of the presentation co-sponsored and hosted by Sogeti since you can see all slides below:

I did wanted to go over some of the questions I was asked at the end of the presentation, since I found them interesting and worth reviewing a little more in detail.

Question: Do you test during the iteration on the cloud or do you test on in-house servers?  When do you test on the cloud, throughout the iteration or only during regression testing?
We believe in testing as soon as possible on servers that closely reflect our production environment, that is why all our tests (sanity, functional, regression, etc) are performed on servers and environments hosted on the cloud and using infrastructure similar (or identical) to the one we use to run our products live.
This is always a good practice, but it is vital for SaaS and cloud-based applications, where even a small issue in the configuration of the servers can make very big differences in the behavior of the system and the user-experience of our customers worldwide.

Question: SaaS applications are typically highly configurable i.e. different clients have different options and pages etc.  How would you approach this in terms of regression testing? Do you pick a sub-set of the most widely used pages, and create a dummy client configuration or what would you suggest as a best approach?
Answer: This is truly one of the challenges of testing a configurable and modular QA Management Solution such as PractiTest where our users not only can choose to use different modules of the system but can also configure it on different ways to match their needs.
I find that the best way to handle this challenge is by working with User Profiles, and having different environments set up ahead of time that reflect real-life configurations and customizations of our users.  This approach allows us to work on testing environments that reflect the whole spectrum of configurations and customizations we may have.
We also use a learning approach where on the occasions a new issue is detected on our Production environments we learn from it and find ways to incorporate it into our testing process, many times this involves adding a customization pattern to one of our existing profiles.

Question: Are there any specific challenges associated with being on the cloud (Amazon in your case) in regards to functional or load & performance testing?
Answer: There were some challenges involved with our initial testing steps in the cloud.  We’ve learned a lot and constantly improve the ways we work as we run into new difficulties and learn from them.  For example, I already said that our testing environments sit on the cloud, at the beginning it was very difficult to update these environments with new builds or even specific patches we wanted to test.  With time we realized we needed “something” that would allow the QA to deploy the product on the testing environment without the constant help of Development or IT; so we worked with them to create a set of scripts that would allows us to deploy to our testing environments automatically.
And this is obviously only an example of the many things we now do differently due to our testing environment constraints.

There were some additional questions that were asked on the webinar, but I will leave them for future posts since I believe they require more detailed attention.

In the meantime I wanted to thanks SoftTest & Anne-Marrie for inviting me, Sogeti for providing the logistical help and the hosting, and obviously all the people who attended for their time and their interesting questions and feedback.

If someone has additional questions from the presentation or the slides you can post them as comments and I will answer them gladly!

, ,

  • Thanks for all the help Michael!
    Will be glad to collaborate in the future.