Bug #12002
closed
async unlinked drain
100%
Description
Each ZFS file system has a list of objects that have become unreferenced, but were not able to be fully deleted from the pool. This list appears to be populated through a number of edge cases; e.g., when a file is unlinked in a file system that is too full to record the deletion (!), or when a file was unreferenced (but still open) at the time of a forced unmount of a file system. The list is stored in a ZAP object called the "ZFS delete queue", but is now more commonly referred to as the "unlinked set".
At mount time, zfs_domount() calls zfsvfs_setup(), and that function calls zfs_unlinked_drain() for a read-write mount. At pool import time, we call zfs_domount() serially for each file system that is automatically mounted, blocking pool import for a considerable period of time.
By way of an anecdote, I recently witnessed a machine take upwards of an hour to import the zones pool. Almost the entire hour was spent iterating the unlinked set, ~10-15 entries per second. Deletion activity from unlinked set processing was inducing 10-15MB/s of disk I/O in a large number of short-lived transaction groups.
It is reasonably clear that synchronously draining the unlinked set at mount time is less than ideal.
This problem has been fixed in ZoL with commit:
dcec0a12c8 port async unlinked drain from illumos-nexenta
This patch is an async implementation of the existing sync zfs_unlinked_drain() function. This function is called at mount time and is responsible for freeing znodes that we didn't get to freeing before. We don't have to hold mounting of the dataset until the unlinked list is fully drained as is done now. Since we can process the unlinked set asynchronously this results in a better user experience when mounting a dataset with entries in the unlinked set.
The ZoL work was in turn based on the following two Nexenta commits:
ea0ae4efc65 NEX-8972 Async-delete side-effect that may cause unmount EBUSY
5d09257a00d NEX-3762 Appliance crashes with a NULL pointer dereference ...
We need to port this work to illumos.
Related issues
Updated by Jerry Jelinek over 3 years ago
The ZoL code added a new test case (umount_unlinked_drain) but that code currently depends on Linux /proc and also other kstat changes which we don't yet have in illumos. I am delivering that test but I did not add it to the run files.
The original Nexenta fix also added a new test case for their change (umount_002). I have included that test case as well since it does run on illumos and I have added that test to the run files.
For testing, I've run the zfs test suite on a DEBUG and non-DEBUG build. This includes the new test (umount_002).
Updated by Electric Monk over 3 years ago
- Status changed from New to Closed
- % Done changed from 0 to 100
git commit c5832a5333c189dfa346a3c1edac9fa39e1de4cb
commit c5832a5333c189dfa346a3c1edac9fa39e1de4cb
Author: Alek Pinchuk <apinchuk@datto.com>
Date: 2019-11-26T21:27:07.000Z
12002 async unlinked drain
Portions contributed by: Jerry Jelinek <jerry.jelinek@joyent.com>
Portions contributed by: Roman Strashkin <roman.strashkin@nexenta.com>
Portions contributed by: Saso Kiselkov <saso.kiselkov@nexenta.com>
Reviewed by: Jorgen Lundman <lundman@lundman.net>
Reviewed by: Tom Caputi <tcaputi@datto.com>
Reviewed by: Brian Behlendorf <behlendorf1@llnl.gov>
Reviewed by: Matt Ahrens <mahrens@delphix.com>
Reviewed by: Paul Dagnelie <pcd@delphix.com>
Reviewed by: Sanjay Nadkarni <sanjay.nadkarni@nexenta.com>
Reviewed by: Josef 'Jeff' Sipek <josef.sipek@nexenta.com>
Reviewed by: C Fraire <cfraire@me.com>
Reviewed by: Kody Kantor <kody.kantor@joyent.com>
Approved by: Dan McDonald <danmcd@joyent.com>
Updated by Andy Fiddaman about 3 years ago
- Related to Bug #12783: async unlinked drain races with ZFS unmount added