Very slow reaping, possible deadlock in zfs_delmap
See original Joyent bug: https://smartos.org/bugview/OS-732
sa_bulk_update call was fixed up in illumos in 2017 (#5379, commit 80e10fd), but the call to
zfs_delmap is still around.
This causes extreme slow-downs on systems reaping many processes with shared mappings of files on ZFS -- which we notice often with thousands of Samba smbd children calling
exit() at once -- the system can process about 1-2 of them exiting per <i>second</i>, meaning that thousands of them take hours to shut down. We have had as many as 14,000 to 18,000 smbd processes on several of our fileservers, which takes 4-5 hours to shut down with that
VOP_PUTPAGE there. Without the
VOP_PUTPAGE it takes minutes.
According to the original analysis in OS-732 by Bryan Cantrill, it seems that this call can also be the source of a deadlock. I haven't personally observed that, so I'm not sure if that's still the case (other things may have changed in ZFS since then).
Quoting from the original bug:
in discussion with the engineer who originally did this work, the consensus is that these semantics are simply too expensive to implement; the fix here is to restore the semantics prior to their change to accommodate the scenario mentioned in the zfs_delmap() comment, above.
Updated by Joshua M. Clulow about 1 year ago
This behaviour appears to have been introduced in:
commit b468a217b67dc26ce21da5d5a2ca09bb6249e4fa Author: eschrock <none@none> Date: Sat Apr 8 23:33:38 2006 -0700 6407791 bringover into ZFS results in s. files newer than extracted source 6409927 failed DKIOCFLUSHWRITECACHE ioctls should not generate ereports 6410371 need to reserve more pool names
Some of the information is available in the historical bug reports; of most relevance here appears to be 6407791
Updated by Electric Monk 12 months ago
- Status changed from New to Closed
- % Done changed from 0 to 100
commit 94cc9d8febd5c99331fd191291b3b54435a1ef18 Author: Alex Wilson <email@example.com> Date: 2020-10-27T23:23:48.000Z 13118 Very slow reaping, possible deadlock in zfs_delmap Portions contributed by: Bryan Cantrill <firstname.lastname@example.org> Reviewed by: Joshua M. Clulow <email@example.com> Approved by: Dan McDonald <firstname.lastname@example.org>