What determines the Run Status of your Tests and Test Sets?
Table of Contents
Test Run Status
- By default, before any steps are run, the status of a Test Run is set to NO RUN.
- If there’s at least one FAILED step, then Test Status is Failed
- If there’s at least one step which is BLOCKED (no FAILED steps), then status is BLOCKED
- Once the Test Run has begun, as long as all the steps you have run up to now are either Passed or N/A, and if you still have steps that you need to run, the status will appear as NOT COMPLETED.
- Only if all steps of the test are defined as PASSED (or some can be also be defined as N/A and then will be disregarded) then the test status will be PASSED as well.
- If all steps are set to N/A, the status of the Test Run will appear as N/A as well.
The Test Run Status that appears in the Test Library reflects the status of the last time that test was run on any test set.
Sometimes it’s much easier to look at the code to understand how it works:
Test Set Run Status
The Run Status of a TestSet is calculated similarly to Tests Run Status:
- By default, the status of a TestSet is NO RUN.
- Once the TestSet has begun running but not finished, and as long as no Test Instances have been set to Blocked or Failed, the TestSet’s status will be NOT COMPLETE.
- If one of the Test Instances is BLOCKED (and there are no instances with status Failed) – the entire TestSet’s Run Status will be BLOCKED as well.
- If one of the Test Instances of a TestSet is FAILED it overrides any previous status (including BLOCKED) and the entire TestSet Status will be set to FAILED.
- Only if all Test Instances of the TestSet are PASSED (or some can be also be defined as N/A and then will be disregarded) then the TestSet status will be PASSED as well.
- If all Test Instances of the TestSet are marked as N/A, the TestSet status will appear as N/A as well.
xBot Test Run Statuses
Here are the optional statuses for an xBot test and their meaning:
- PASSED - The test ran and the result is 0
- FAILED - The test ran and the result is anything but 0
- RUNNING - The test run was taken by an agent. The agent is now running the user script on the user machine.
- PENDING - The user hit the run button (on the set/instance/run), the run was added to the runs queue. the run is now waiting for an agent with a matching client type to be available to run/execute this run (if an agent is not picking up the test, it is either because the agent is busy OR the client type does not match OR the agent is disabled OR the run is scheduled for later)
- CANCELED - The execution was canceled while it was running. (please see below)
- ERROR - The xBot agant failed to execute the test and the run/script execution was not completed. For example when the script doesn't exist, or is not executable