What’s the deal with regression testing and why are we still spending so much time on it?

How’s life

Joel: 

  • December is great, I love December!  I get to stay in Israel and work from the office as most of our customers are busy closing the year and getting ready for Christmas.
  • Planning next year, many trips and conferences.
  • It will be great year, but right now trying to regroup and prepare for it!

Rob:

  • Getting set for Christmas

Regression Testing

  • Still a huge focus at conferences, why?
  • Are we still struggling to automate it all as the vendors promised? 
  • Do we need to do it?
  • What’s the purpose?
  • If regression testing doesn’t find anything – was it useful?
  • How to pick regression tests?

Regression testing is needed!

  • Not every bug is bad, some bugs are like trees falling on empty forests.  No one really cares about them. But let’s leave them aside.
  • Obviously some bugs are disastrous on their own, but if they are not and if you have 2 bugs that are equally bad, the one that is a new piece of functionality might be more acceptable than the one that is on a piece of functionality that has been out for a long time and was stable up to the last release.
  • Many organizations and system flag regression bugs specifically because of this, to make sure our stable functionality remains stable.  And what’s more, if it is a bug that was already reported and fixed, if it is re-released, it is even worse!!!
  • Regression testing can be defined in many ways, but in layman terms, it is testing everything to make sure no surprises were introduced without notice.  Especially in areas that were not supposed to change.
  • The main assumption is that a regression test should not find anything, but when they do these are usually very important bugs.
  • If it didn’t find anything, over and over again, is it worth running? 
    • Should it all be automated – we’ll talk about that later.

Not every regression test was created equally

  • Full regression, sanity, smoke.
  • Automatic, semi-automatic.
  • Specific regression (one case I remember is database migration data-driven regression).
  • How big can your regression pack go?
  • Can regression tests be exploratory?  Trick question.

Automatic regression?

  • Should all regression be automated?
  • Do we need to limit the number of tests in our automated regression suite?
  • Do we run all the regression as part of our CI?

Is regression the last test before releasing in order to find regression bugs?

  • Bug bashes or Bug hunts.
  • Dogfooding rules!
  • Alpha and Beta tests.

Is testing in production regression testing?

YES!

One Response to What’s the deal with regression testing and why are we still spending so much time on it?

  1. Alexei February 4, 2020 at 1:43 pm #

    Hi,

    About “Full regression, sanity, smoke” — do you have different definitions for sanity and smoke testing?

    Thank you.

Leave a Reply