Project

General

Profile

Feature #11814

test-runner exit code should reflect test results

Added by Tim Foster 4 months ago. Updated 3 months ago.

Status:
Closed
Priority:
Normal
Assignee:
Category:
tests
Start date:
Due date:
% Done:

100%

Estimated time:
Difficulty:
Medium
Tags:

Description

With the changes introduced to SmartOS with https://smartos.org/bugview/OS-7943, we use /opt/test-runner/bin/run to execute tests built by the illumos-gate.

We noticed that despite the presence of failed or killed tests, the test-runner script always exits 0.

To avoid having to parse the text output of the runner, we'd like to add a '-e' flag to the test-runner to have it exit non-zero at the end of the test run if any of the tests report a 'FAIL' or 'KILLED' state.

The default would be to continue with the existing behaviour, to prevent impacting possibly existing automation.

History

#1

Updated by Tim Foster 4 months ago

  • Subject changed from https://smartos.org/bugview/OS-7943 to test-runner should have an option to exit non-zero when tests fail
#2

Updated by Tim Foster 4 months ago

  • Subject changed from test-runner should have an option to exit non-zero when tests fail to test-runner should require an option to exit zero when tests fail
#3

Updated by Tim Foster 4 months ago

Checking will the illumos-developer list on what exactly the default should be:

  • require a '-e' to cause the script to exit non-zero in the case of failure, which has the benefit of not breaking existing consumers,
  • change the default so that '-e' is needed to cause the script to exit 0, even when tests have failed.
#4

Updated by John Levon 4 months ago

I don't really understand why we'd ever exit 0 on failure?

#5

Updated by Tim Foster 4 months ago

The only reason I had to preserve the functionality was that existing distro automation might expect it.

One could imagine test automation that might expect non-zero exits to only indicate failing test framework code, rather than indicating failed tests, which might otherwise be detected after tests complete by parsing test logs.

I don't know if other distros have such expectations - if not, I'm fine with removing the flag altogether.

#6

Updated by John Levon 3 months ago

  • Subject changed from test-runner should require an option to exit zero when tests fail to test-runner exit code should reflect test results

We'll just always return 1 if we hit a FAIL or 2 if a test got killed due to timeout

#7

Updated by John Levon 3 months ago

  • Assignee changed from Tim Foster to John Levon
#8

Updated by John Levon 3 months ago

I tested this passed on the success or failure as needed by making a test fail,
and also checked all the callers pass it on as needed too.

#9

Updated by Electric Monk 3 months ago

  • Status changed from New to Closed
  • % Done changed from 0 to 100

git commit 2491fc01733d7e4f5776b0711db713078d7cd24a

commit  2491fc01733d7e4f5776b0711db713078d7cd24a
Author: John Levon <john.levon@joyent.com>
Date:   2019-11-16T11:20:54.000Z

    11814 test-runner exit code should reflect test results
    Reviewed by: Robert Mustacchi <rm@fingolfin.org>
    Approved by: Dan McDonald <danmcd@joyent.com>

Also available in: Atom PDF