Add option to testrunner to re-run just failed tests
It would be really handy if
testrunner supported an option to only run tests which failed on the last run.
Currently one has to copy and modify the run file to remove the tests which are no longer of interest and that's quite a tedious process with the larger test suites.
Updated by Andy Fiddaman over 1 year ago
This patch also allows the creation of a new run file with just the failing tests and supporting framework for them by combining the
-w options. Here's an example:
% fgrep '[FAIL]' /var/tmp/test_results/20201214T103509/log Test: /opt/zfs-tests/tests/functional/acl/nontrivial/zfs_acl_chmod_inherit_003_pos (run as root) [01:46] [FAIL] Test: /opt/zfs-tests/tests/functional/cli_root/zpool_import/setup (run as root) [00:04] [FAIL] Test: /opt/zfs-tests/tests/functional/vdev_zaps/vdev_zaps_007_pos (run as root) [00:07] [FAIL]
% /opt/test-runner/bin/run -c /opt/zfs-tests/runfiles/omnios.run -l /var/tmp/test_results/20201214T103509/log -w new.run
% cat new.run [DEFAULT] pre = quiet = False pre_user = user = timeout = 60 post_user = post = outputdir = /var/tmp/test_results [/opt/zfs-tests/tests/functional/acl/nontrivial] tests = ['zfs_acl_chmod_inherit_003_pos'] timeout = 1800 user = root pre = /opt/zfs-tests/tests/functional/acl/nontrivial/setup pre_user = root post = /opt/zfs-tests/tests/functional/acl/nontrivial/cleanup post_user = root [/opt/zfs-tests/tests/functional/cli_root/zpool_import] tests = ['zpool_import_001_pos', 'zpool_import_002_pos', 'zpool_import_003_pos', 'zpool_import_004_pos', 'zpool_import_005_pos', 'zpool_import_006_pos', 'zpool_import_007_pos', 'zpool_import_008_pos', 'zpool_import_009_neg', 'zpool_import_010_pos', 'zpool_import_011_neg', 'zpool_import_012_pos', 'zpool_import_013_neg', 'zpool_import_014_pos', 'zpool_import_all_001_pos', 'zpool_import_features_001_pos', 'zpool_import_features_002_neg', 'zpool_import_features_003_pos', 'zpool_import_missing_001_pos', 'zpool_import_missing_002_pos', 'zpool_import_missing_003_pos', 'zpool_import_rename_001_pos', 'import_cachefile_device_added', 'import_cachefile_device_removed', 'import_cachefile_mirror_attached', 'import_cachefile_mirror_detached', 'import_cachefile_device_replaced', 'import_rewind_config_changed', 'import_cachefile_shared_device', 'import_paths_changed', 'import_devices_missing', 'zpool_import_encrypted', 'zpool_import_encrypted_load'] timeout = 1800 user = root pre = /opt/zfs-tests/tests/functional/cli_root/zpool_import/setup pre_user = root post = /opt/zfs-tests/tests/functional/cli_root/zpool_import/cleanup post_user = root [/opt/zfs-tests/tests/functional/vdev_zaps] tests = ['vdev_zaps_007_pos'] timeout = 1800 user = root pre = /opt/zfs-tests/tests/functional/vdev_zaps/setup pre_user = root post = /opt/zfs-tests/tests/functional/vdev_zaps/cleanup post_user = root
You can see from this that it properly detects when a test group pre-requisite failed and leaves the full list of dependent tests enabled.
Updated by Electric Monk over 1 year ago
- Status changed from In Progress to Closed
- % Done changed from 0 to 100
commit 78801af7286cd73dbc996d470f789e75993cf15d Author: Andy Fiddaman <email@example.com> Date: 2021-01-26T21:14:19.000Z 13365 Add option to testrunner to re-run just failed tests Reviewed by: Jason King <firstname.lastname@example.org> Approved by: Robert Mustacchi <email@example.com>