Project

General

Profile

Actions

Feature #13365

closed

Add option to testrunner to re-run just failed tests

Added by Andy Fiddaman over 1 year ago. Updated over 1 year ago.

Status:
Closed
Priority:
Normal
Assignee:
Category:
tests
Start date:
Due date:
% Done:

100%

Estimated time:
Difficulty:
Medium
Tags:
Gerrit CR:

Description

It would be really handy if testrunner supported an option to only run tests which failed on the last run.
Currently one has to copy and modify the run file to remove the tests which are no longer of interest and that's quite a tedious process with the larger test suites.

Actions #1

Updated by Andy Fiddaman over 1 year ago

This patch also allows the creation of a new run file with just the failing tests and supporting framework for them by combining the -l and -w options. Here's an example:

  % fgrep '[FAIL]' /var/tmp/test_results/20201214T103509/log
  Test: /opt/zfs-tests/tests/functional/acl/nontrivial/zfs_acl_chmod_inherit_003_pos (run as root) [01:46] [FAIL]
  Test: /opt/zfs-tests/tests/functional/cli_root/zpool_import/setup (run as root) [00:04] [FAIL]
  Test: /opt/zfs-tests/tests/functional/vdev_zaps/vdev_zaps_007_pos (run as root) [00:07] [FAIL]
  % /opt/test-runner/bin/run -c /opt/zfs-tests/runfiles/omnios.run -l /var/tmp/test_results/20201214T103509/log -w new.run
  % cat new.run
  [DEFAULT]
  pre =
  quiet = False
  pre_user =
  user =
  timeout = 60
  post_user =
  post =
  outputdir = /var/tmp/test_results

  [/opt/zfs-tests/tests/functional/acl/nontrivial]
  tests = ['zfs_acl_chmod_inherit_003_pos']
  timeout = 1800
  user = root
  pre = /opt/zfs-tests/tests/functional/acl/nontrivial/setup
  pre_user = root
  post = /opt/zfs-tests/tests/functional/acl/nontrivial/cleanup
  post_user = root

  [/opt/zfs-tests/tests/functional/cli_root/zpool_import]
  tests = ['zpool_import_001_pos', 'zpool_import_002_pos', 'zpool_import_003_pos', 'zpool_import_004_pos', 'zpool_import_005_pos', 'zpool_import_006_pos', 'zpool_import_007_pos', 'zpool_import_008_pos', 'zpool_import_009_neg', 'zpool_import_010_pos', 'zpool_import_011_neg', 'zpool_import_012_pos', 'zpool_import_013_neg', 'zpool_import_014_pos', 'zpool_import_all_001_pos', 'zpool_import_features_001_pos', 'zpool_import_features_002_neg', 'zpool_import_features_003_pos', 'zpool_import_missing_001_pos', 'zpool_import_missing_002_pos', 'zpool_import_missing_003_pos', 'zpool_import_rename_001_pos', 'import_cachefile_device_added', 'import_cachefile_device_removed', 'import_cachefile_mirror_attached', 'import_cachefile_mirror_detached', 'import_cachefile_device_replaced', 'import_rewind_config_changed', 'import_cachefile_shared_device', 'import_paths_changed', 'import_devices_missing', 'zpool_import_encrypted', 'zpool_import_encrypted_load']
  timeout = 1800
  user = root
  pre = /opt/zfs-tests/tests/functional/cli_root/zpool_import/setup
  pre_user = root
  post = /opt/zfs-tests/tests/functional/cli_root/zpool_import/cleanup
  post_user = root

  [/opt/zfs-tests/tests/functional/vdev_zaps]
  tests = ['vdev_zaps_007_pos']
  timeout = 1800
  user = root

  pre = /opt/zfs-tests/tests/functional/vdev_zaps/setup
  pre_user = root

  post = /opt/zfs-tests/tests/functional/vdev_zaps/cleanup
  post_user = root

You can see from this that it properly detects when a test group pre-requisite failed and leaves the full list of dependent tests enabled.

Actions #2

Updated by Electric Monk over 1 year ago

  • Gerrit CR set to 1103
Actions #3

Updated by Andy Fiddaman over 1 year ago

I've been using this regularly for a few weeks to iteratively re-run just the failing ZFS tests until I get a clean (enough) set.

Actions #4

Updated by Electric Monk over 1 year ago

  • Status changed from In Progress to Closed
  • % Done changed from 0 to 100

git commit 78801af7286cd73dbc996d470f789e75993cf15d

commit  78801af7286cd73dbc996d470f789e75993cf15d
Author: Andy Fiddaman <omnios@citrus-it.co.uk>
Date:   2021-01-26T21:14:19.000Z

    13365 Add option to testrunner to re-run just failed tests
    Reviewed by: Jason King <jason.king@joyent.com>
    Approved by: Robert Mustacchi <rm@fingolfin.org>

Actions

Also available in: Atom PDF