poll(2) returns prematurely in presence of spurious wakeups
From Matt Ahrens's original bug report at Delphix (the deadman thing he mentions hasn't been pushed to Illumos yet, but the bug still exists):
poll(2) is supposed to wait for the entire time requested, unless a requested
event happens or a signal is delivered. However, in the presence of many
spurious wakeups, it will wait for much less than the time requested. Each
wakeup reduces the amount of time waited by up to 1 tick (10ms).
I found this because ztest_deadman_thread() uses poll(2) to wait for 360
seconds, but poll() returns after less than 60 seconds, causing the deadman to fire and ztest to exit.
Watchpoints cause many spurious wakeups, because many watchpoint actions (e.g. installing a new watchpoint from pr_watch()) call pokelwps(), which forces all threads into the kernel, waking up T_WAKEABLE threads, including the ones in cv_*wait*sig*.
poll() uses cv_timedwait_sig_hires() to wait, which returns the number of ticks remaining to wait. poll_common() calls cv_timedwait_sig_hires() in a loop, each time waiting the amount of time returned by the previous call. However, since the time remaining is returned in ticks, it goes down by at least one tick each call.
Therefore, if there are many spurious wakeups (because we are calling the shit out of pr_watch()), poll() will call cv_timedwait_sig_hires() "ticks" times,
each time waiting for much less than a tick (about 1ms). In our use case this
means that ztest() will fire its deadman prematurely if used with watchpoints.
1605 System time changes combined with spurios wakeups cause DP_POLL to return prematurely
The fix is to return this code to approximately the original logic, where we
compute the absolute deadline time and wait until then. However, we will use a lbolt-based deadline, rather than wall-clock time. Since lbolt only marches forward, we don't have to worry about the system clock being changed.
Updated by Eric Schrock over 7 years ago
- Status changed from In Progress to Resolved
user: Matthew Ahrens <email@example.com>
date: Thu Aug 30 05:13:49 2012 -0700
3112 ztest does not honor ZFS_DEBUG
3113 ztest should use watchpoints to protect frozen arc bufs
3114 some leaked nvlists in zfsdev_ioctl
3115 poll(2) returns prematurely in presence of spurious wakeups
Reviewed by: Adam Leventhal <firstname.lastname@example.org>
Reviewed by: Matt Amdur <Matt.Amdur@delphix.com>
Reviewed by: George Wilson <email@example.com>
Reviewed by: Christopher Siden <firstname.lastname@example.org>
Approved by: Eric Schrock <email@example.com>