The following is a guest post by Ali Hill, Delivery Consultant, at ECS.
The industry’s move to accelerated releases, new tools and techniques can pose a challenge to testers. There is a lot to learn and often a lot of pressure to do so quickly. It’s not uncommon for testers to become a bottleneck in DevOps environments and it’s something I’ve experienced myself. There is nothing more frustrating than trying to add value to a team, but at the same time, temporarily blocking the team’s release process.
The definition of Continuous Testing in the DevOps field, and the industry as a whole, focuses almost exclusively on test automation. It describes the tests that take place in a CI/CD pipeline to provide fast feedback to the team. I believe that this definition of Continuous Testing is far too narrow and doesn’t reflect a lot of the ways in which testers can provide value to teams. Automated tests are vital for any team wanting to release frequently, but there are many other roles that testers play in DevOps teams.
Get Involved in Gathering Requirements
It’s important that, as a team, we know what it is we’re building and why. What problem are we trying to solve with this new piece of functionality? These questions are best asked at a Three Amigos session where a product person, tester and developer are all present. It is vital that the whole team is involved in these discussions.
Once you understand the problem you’re trying to solve, you can use examples to try and find any missing requirements and promote a shared understanding across the team. This is an area that testers excel in. We are great at thinking outside of the box when it comes to testing functionality and we can apply the same thought process at these early stages. Think of examples of how the user will use your new functionality and bring these to the team.
For example, if you’re designing password reset functionality then the main path is to allow users to reset their password. You may ask “can the user reset their password to a previously used password?” or “is there a limit to the number of times a user can reset their password in a day?” These types of questions help to promote a shared understanding across the team.
By discussing these scenarios upfront, you’re reducing the likelihood that anyone has misunderstood the goal of this new piece of functionality. It’s much easier and cheaper to discuss disagreements or misunderstandings at this stage than it is after the functionality has already been deployed to a test environment.
Share Test Scenarios With Your Team
Now that you’ve got a shared understanding of the functionality you want to develop, you can start to talk about test scenarios with your team. Using our previous example, if you’re designing a password reset system, and the product owner decides you cannot use one of your previous 5 passwords when resetting your password, then that becomes a test at least one layer of your application.
The value of sharing the tests that you want to run is that it reduces the likelihood of unexpected “bugs” being raised further down the pipeline. If you can say to your team “I plan on running these tests, and these are my expected results” then you are ensuring that everyone knows what it is you’re looking for in the behavior of the functionality.
Your team doesn’t have to agree with you, the product owner or developer may disagree on the expected results, but it’s important to have these conversations as early as possible as it can lead to a reduction in wasted time and effort. If these conversations happen after automated tests are run in a pipeline, for example, you’re having to do a lot more re-work than if the issue is caught early.
In my experience, it is important to get these sessions occurring frequently in a DevOps environment, where frequent releases are occurring. The more effort you put into understanding requirements, the less re-work you will have to do. Sometimes things as simple as a shared understanding of what you’re developing can accelerate the rate at which you can develop and release quality software.
Pair on Writing Unit Tests
Writing unit tests or practicing TDD is often seen as a developer’s activity and isn’t something I have seen a lot of testers getting involved in. Based on my experiences, this is an opportunity lost.
It’s important to state that TDD is a design activity. The unit tests that are written during the TDD process are used to guide a developer’s implementation and how they organize their code. But where does the information for which tests to write come from?
The tests that are being written should be informed by the requirements you had previously discussed with your team. I’ve found it extremely valuable as a tester to work with developers as they practice TDD or write unit tests.
It’s a great opportunity to reinforce the understanding of the requirements that you and your team have. If you’re sitting with a developer and they’re writing unit tests, as a tester, you should be able to identify why they are writing those specific tests. Keep the requirements handy and refer to them often.
Pairing on writing unit tests is valuable for both roles. As testers, we’re good at keeping a focus on the customer and requirements, whilst the developer we’re working with is getting stuck into the code.
It also has a second benefit. You can begin to talk about the testability of the code under development. Of course, practicing TDD means that the developer’s code should be testable at a unit level. But if you have discussed some UI or API tests that you wish to run, then this provides the opportunity to implement exception handling, expected response formatting and hooks into the UI for automated UI testing.
If you’re the one writing the tests – you know exactly what to look into and can begin writing these tests right away. If the developer is writing the tests, then they’re creating a testable implementation for themselves.
One of the main advantages of practicing TDD is that you end up with a large suite of tests that can be run every time code is committed. Typically what you will find with unit tests is that they test the happy path. But what about the edge or boundary cases?
If we refer back to the reset password example, what if someone were to want to re-use their 5th most recent password? It shouldn’t be allowed, but has our code handled an off by 1 error? If it is correctly not allowed then are the correct exceptions being thrown?
It is these types of scenarios that testers can add a lot of value within TDD or with writing unit tests. Why wait until the build has been deployed to a test environment to catch issues like this? If a tester is involved at this stage, you’ll generally find that cases like this are caught earlier, thus preventing re-work further down the line. It’s a great way to exploratory test the application at an early stage of development.
In a DevOps environment, we are not finished with a piece of functionality until we have proven that it runs successfully in production. Once that has been proven, and the functionality is in front of customers, we can begin to see how they’re using it. Observability needs to be discussed at the early stages of development, but the logging we are outputting to trace our users’ actions and data being sent through pipelines, etc. means we can observe what is happening in production.
We should be validating that what we have developed is being used by the customer, the way we would expect. As well as the product owner or the team getting feedback from customers directly, we can do this through various monitoring tools. It is this data that allows us to prove that we have solved a problem for the customer.
As testers, we have traditionally acted as an advocate for the customer. Monitoring tools should make our life easier in terms of observing quantitative data to prove that what we have released is working, both without error and in solving a customer’s problem. We can also get involved in setting up alerts for when errors do occur in production.
Given the complex nature of most of the applications we work on now, testing in production is becoming a necessity for a lot of teams. We can set up test or staging environments to look as much like production as possible, but each of these environments will always be unique, due to data, the activity taking place on the environment or even small differences in the way it is configured. How many times have you seen a bug occur in production that didn’t occur in your test environment?
Start off by running as many non-disruptive tests as possible in your production environment. Health checks that ensure a basic GET on your endpoints or basic UI checks are useful here. You can run these checks until your team feels more confident running tests in production and then begin to expand. Make the results of these tests easily accessible to all so that everyone knows the health status of your production environment.
The role of testers in DevOps environments is to grow in our role to add value wherever possible in our teams. We should be redefining what it means to be a tester by getting involved at all stages of the development process. Collaborate with the team when gathering requirements and writing unit tests to ensure any issues are caught or prevented as early as possible. Use tools, such as monitoring solutions, to ensure that our solutions are high quality and providing value to our customers.
Of course, automated tests are extremely important. Especially in complex solutions. They allow us to identify if a change has been made to our critical functionality by providing fast feedback. But there are so many other ways that testers can provide value in a DevOps environment, and I’ve outlined some of these in this article.
Ali began his software testing career testing video games before moving into an Agile testing role. He now works as a Delivery Consultant, specializing in DevOps, at ECS. He has a passion for learning. Recently Ali has been interested in learning how software testing can flourish in the world of DevOps and Continuous Delivery. Ali can be found talking about testing on Twitter, blogging or at the Edinburgh Ministry of Testing Meetup
Practitest is an end-to-end test management tool, that gives you control of the entire testing process - from manual testing to automated testing and CI.
Designed for testers by testers, PractiTest can be customized to your team's ever-changing needs.
With fast professional and methodological support, you can make the most of your time and release products quickly and successfully to meet your user’s needs.