Project

General

Profile

Actions

Feature #13365

closed

Add option to testrunner to re-run just failed tests

Added by Andy Fiddaman 12 months ago. Updated 10 months ago.

Status:
Closed
Priority:
Normal
Assignee:
Category:
tests
Start date:
Due date:
% Done:

100%

Estimated time:
Difficulty:
Medium
Tags:
Gerrit CR:

Description

It would be really handy if testrunner supported an option to only run tests which failed on the last run.
Currently one has to copy and modify the run file to remove the tests which are no longer of interest and that's quite a tedious process with the larger test suites.

Actions #1

Updated by Andy Fiddaman 12 months ago

This patch also allows the creation of a new run file with just the failing tests and supporting framework for them by combining the -l and -w options. Here's an example:

  % fgrep '[FAIL]' /var/tmp/test_results/20201214T103509/log
  Test: /opt/zfs-tests/tests/functional/acl/nontrivial/zfs_acl_chmod_inherit_003_pos (run as root) [01:46] [FAIL]
  Test: /opt/zfs-tests/tests/functional/cli_root/zpool_import/setup (run as root) [00:04] [FAIL]
  Test: /opt/zfs-tests/tests/functional/vdev_zaps/vdev_zaps_007_pos (run as root) [00:07] [FAIL]
  % /opt/test-runner/bin/run -c /opt/zfs-tests/runfiles/omnios.run -l /var/tmp/test_results/20201214T103509/log -w new.run
  % cat new.run
  [DEFAULT]
  pre =
  quiet = False
  pre_user =
  user =
  timeout = 60
  post_user =
  post =
  outputdir = /var/tmp/test_results

  [/opt/zfs-tests/tests/functional/acl/nontrivial]
  tests = ['zfs_acl_chmod_inherit_003_pos']
  timeout = 1800
  user = root
  pre = /opt/zfs-tests/tests/functional/acl/nontrivial/setup
  pre_user = root
  post = /opt/zfs-tests/tests/functional/acl/nontrivial/cleanup
  post_user = root

  [/opt/zfs-tests/tests/functional/cli_root/zpool_import]
  tests = ['zpool_import_001_pos', 'zpool_import_002_pos', 'zpool_import_003_pos', 'zpool_import_004_pos', 'zpool_import_005_pos', 'zpool_import_006_pos', 'zpool_import_007_pos', 'zpool_import_008_pos', 'zpool_import_009_neg', 'zpool_import_010_pos', 'zpool_import_011_neg', 'zpool_import_012_pos', 'zpool_import_013_neg', 'zpool_import_014_pos', 'zpool_import_all_001_pos', 'zpool_import_features_001_pos', 'zpool_import_features_002_neg', 'zpool_import_features_003_pos', 'zpool_import_missing_001_pos', 'zpool_import_missing_002_pos', 'zpool_import_missing_003_pos', 'zpool_import_rename_001_pos', 'import_cachefile_device_added', 'import_cachefile_device_removed', 'import_cachefile_mirror_attached', 'import_cachefile_mirror_detached', 'import_cachefile_device_replaced', 'import_rewind_config_changed', 'import_cachefile_shared_device', 'import_paths_changed', 'import_devices_missing', 'zpool_import_encrypted', 'zpool_import_encrypted_load']
  timeout = 1800
  user = root
  pre = /opt/zfs-tests/tests/functional/cli_root/zpool_import/setup
  pre_user = root
  post = /opt/zfs-tests/tests/functional/cli_root/zpool_import/cleanup
  post_user = root

  [/opt/zfs-tests/tests/functional/vdev_zaps]
  tests = ['vdev_zaps_007_pos']
  timeout = 1800
  user = root

  pre = /opt/zfs-tests/tests/functional/vdev_zaps/setup
  pre_user = root

  post = /opt/zfs-tests/tests/functional/vdev_zaps/cleanup
  post_user = root

You can see from this that it properly detects when a test group pre-requisite failed and leaves the full list of dependent tests enabled.

Actions #2

Updated by Electric Monk 12 months ago

  • Gerrit CR set to 1103
Actions #3

Updated by Andy Fiddaman 10 months ago

I've been using this regularly for a few weeks to iteratively re-run just the failing ZFS tests until I get a clean (enough) set.

Actions #4

Updated by Electric Monk 10 months ago

  • Status changed from In Progress to Closed
  • % Done changed from 0 to 100

git commit 78801af7286cd73dbc996d470f789e75993cf15d

commit  78801af7286cd73dbc996d470f789e75993cf15d
Author: Andy Fiddaman <omnios@citrus-it.co.uk>
Date:   2021-01-26T21:14:19.000Z

    13365 Add option to testrunner to re-run just failed tests
    Reviewed by: Jason King <jason.king@joyent.com>
    Approved by: Robert Mustacchi <rm@fingolfin.org>

Actions

Also available in: Atom PDF