Feature #11814
closed
test-runner exit code should reflect test results
100%
Description
With the changes introduced to SmartOS with https://smartos.org/bugview/OS-7943, we use /opt/test-runner/bin/run to execute tests built by the illumos-gate.
We noticed that despite the presence of failed or killed tests, the test-runner script always exits 0.
To avoid having to parse the text output of the runner, we'd like to add a '-e' flag to the test-runner to have it exit non-zero at the end of the test run if any of the tests report a 'FAIL' or 'KILLED' state.
The default would be to continue with the existing behaviour, to prevent impacting possibly existing automation.
Updated by Tim Foster almost 4 years ago
- Subject changed from https://smartos.org/bugview/OS-7943 to test-runner should have an option to exit non-zero when tests fail
Updated by Tim Foster almost 4 years ago
- Subject changed from test-runner should have an option to exit non-zero when tests fail to test-runner should require an option to exit zero when tests fail
Updated by Tim Foster almost 4 years ago
Checking will the illumos-developer list on what exactly the default should be:
- require a '-e' to cause the script to exit non-zero in the case of failure, which has the benefit of not breaking existing consumers,
- change the default so that '-e' is needed to cause the script to exit 0, even when tests have failed.
Updated by John Levon almost 4 years ago
I don't really understand why we'd ever exit 0 on failure?
Updated by Tim Foster almost 4 years ago
The only reason I had to preserve the functionality was that existing distro automation might expect it.
One could imagine test automation that might expect non-zero exits to only indicate failing test framework code, rather than indicating failed tests, which might otherwise be detected after tests complete by parsing test logs.
I don't know if other distros have such expectations - if not, I'm fine with removing the flag altogether.
Updated by John Levon almost 4 years ago
- Subject changed from test-runner should require an option to exit zero when tests fail to test-runner exit code should reflect test results
We'll just always return 1 if we hit a FAIL or 2 if a test got killed due to timeout
Updated by John Levon almost 4 years ago
- Assignee changed from Tim Foster to John Levon
Updated by John Levon almost 4 years ago
I tested this passed on the success or failure as needed by making a test fail,
and also checked all the callers pass it on as needed too.
Updated by Electric Monk almost 4 years ago
- Status changed from New to Closed
- % Done changed from 0 to 100
git commit 2491fc01733d7e4f5776b0711db713078d7cd24a
commit 2491fc01733d7e4f5776b0711db713078d7cd24a
Author: John Levon <john.levon@joyent.com>
Date: 2019-11-16T11:20:54.000Z
11814 test-runner exit code should reflect test results
Reviewed by: Robert Mustacchi <rm@fingolfin.org>
Approved by: Dan McDonald <danmcd@joyent.com>