Project

General

Profile

Actions

Feature #14358

closed

Cron should support randomized delay

Added by Sebastian Wiedenroth 5 months ago. Updated 4 months ago.

Status:
Closed
Priority:
Normal
Category:
cmd - userland programs
Start date:
Due date:
% Done:

100%

Estimated time:
Difficulty:
Medium
Tags:
Gerrit CR:

Description

Having a large number of cronjobs start at the same time can lead to load spikes.
This is especially noticeable when many zones are running with the same configuration.

Not all jobs need precise timing and having a way to specify a randomized delay can help provide a more even load.

Other systems provide similar features, for example systemd timers have a "RandomizedDelaySec" option.

Actions #1

Updated by Joshua M. Clulow 5 months ago

Is there another popular cron implementation that already provides a syntax for this? If so, it seems like it'd be good to avoid inventing another.

Actions #2

Updated by Electric Monk 5 months ago

  • Gerrit CR set to 1912
Actions #3

Updated by Sebastian Wiedenroth 5 months ago

Here is what I found just now:

I have a draft up on gerrit that uses RANDOMIZED_DELAY in seconds. Happy to either change it to RANDOM_DELAY in minutes or to something like JITTER if we want to keep a way to specify seconds. Any preferences?

Actions #4

Updated by Sebastian Wiedenroth 5 months ago

I've updated the CR to use RANDOM_DELAY in minutes to match the cronie syntax.

Actions #5

Updated by Sebastian Wiedenroth 5 months ago

Testing:

I copied the cron + crontab binaries into place, restarted cron and added the following crontab entries:

* * * * * python -c 'from datetime import datetime; print(datetime.now())' >> /tmp/cron-none.log
RANDOM_DELAY=0
* * * * * python -c 'from datetime import datetime; print(datetime.now())' >> /tmp/cron-0.log
RANDOM_DELAY=1
* * * * * python -c 'from datetime import datetime; print(datetime.now())' >> /tmp/cron-1.log
RANDOM_DELAY=2
* * * * * python -c 'from datetime import datetime; print(datetime.now())' >> /tmp/cron-2.log

The logfiles confirm that the expected delay is used:

==> /tmp/cron-none.log <==
2022-01-09 15:27:00.371923
2022-01-09 15:28:00.423843
2022-01-09 15:29:00.497000
2022-01-09 15:30:00.549232
2022-01-09 15:31:00.626922
2022-01-09 15:32:00.705032
2022-01-09 15:33:00.757227
2022-01-09 15:34:00.834806
2022-01-09 15:35:00.881519
2022-01-09 15:36:00.960147

==> /tmp/cron-0.log <==
2022-01-09 15:27:00.371923
2022-01-09 15:28:00.423879
2022-01-09 15:29:00.496990
2022-01-09 15:30:00.549232
2022-01-09 15:31:00.626864
2022-01-09 15:32:00.705005
2022-01-09 15:33:00.757234
2022-01-09 15:34:00.834862
2022-01-09 15:35:00.878436
2022-01-09 15:36:00.960147

==> /tmp/cron-1.log <==
2022-01-09 15:26:55.345657
2022-01-09 15:27:55.397663
2022-01-09 15:28:44.470520
2022-01-09 15:29:16.522952
2022-01-09 15:30:44.600610
2022-01-09 15:31:08.653651
2022-01-09 15:32:28.731371
2022-01-09 15:33:12.808530
2022-01-09 15:34:33.854378
2022-01-09 15:35:22.911618

==> /tmp/cron-2.log <==
2022-01-09 15:20:06.903962
2022-01-09 15:22:15.064165
2022-01-09 15:23:16.119293
2022-01-09 15:24:07.196130
2022-01-09 15:26:04.320180
2022-01-09 15:28:24.443297
2022-01-09 15:30:37.575247
2022-01-09 15:31:10.679027
2022-01-09 15:33:02.783068
2022-01-09 15:35:57.929780

Providing invalid parameters for the RANDOMIZED_DELAY option is handled as expected:

Invalid delay: RANDOM_DELAY=-1
crontab: errors detected in input, no crontab file generated.

Invalid delay: RANDOM_DELAY=foo
crontab: errors detected in input, no crontab file generated.

Man page was looked at with

man -M  usr/src/man/ crontab.1
.

Actions #6

Updated by Sebastian Wiedenroth 4 months ago

Also more complex crontab entries still work as expected

*/2 * * * * python -c 'from datetime import datetime; print(datetime.now())' >> /tmp/cron-every-two-mins-no-random.log
2022-01-29 17:52:00.439984
2022-01-29 17:54:00.470712
2022-01-29 17:56:00.501591
2022-01-29 17:58:00.531602
2022-01-29 18:00:00.561686
2022-01-29 18:02:00.592390
2022-01-29 18:04:00.623266
2022-01-29 18:06:00.653606
2022-01-29 18:08:00.684327
2022-01-29 18:10:00.717579
Actions #7

Updated by Electric Monk 4 months ago

  • % Done changed from 80 to 100
  • Status changed from New to Closed

git commit 618372bccd696950e1d234d0ad9c94c353882dee

commit  618372bccd696950e1d234d0ad9c94c353882dee
Author: Sebastian Wiedenroth <wiedi@frubar.net>
Date:   2022-01-29T21:05:44.000Z

    14358 Cron should support randomized delay
    Reviewed by: Andy Fiddaman <andy@omnios.org>
    Approved by: Robert Mustacchi <rm@fingolfin.org>

Actions

Also available in: Atom PDF