Project

General

Profile

Actions

Bug #14519

closed

zfs_root should cache vnode

Added by Patrick Mooney about 2 years ago. Updated over 1 year ago.

Status:
Closed
Priority:
Normal
Category:
zfs - Zettabyte File System
Start date:
Due date:
% Done:

100%

Estimated time:
Difficulty:
Medium
Tags:
Gerrit CR:
External Bug:

Description

Partaking in my pastime of dtracing a node while it's performing an illumos-gate build, I found some lockstat(1) data which seemed to warrant additional investigation. A top contender in the adaptive-mutex-blocked category was a lock being acquired in the zfs_zget function. Further investigation lead me to the most common stack (accounting for time spent blocked on the lock):

              zfs`zfs_zget+0x62
              zfs`zfs_root+0x60
              genunix`fsop_root+0x33
              genunix`traverse+0x90
              genunix`lookuppnvp+0x2d9
              genunix`lookuppnatcred+0x134
              genunix`lookuppn+0x2e
              genunix`dogetcwd+0x15f
              genunix`getcwd+0x9b
              unix`sys_syscall+0x1a8
       8035755520

It seems that every time we traverse a mount point, we ask the underlying filesystem for its root vnode. ZFS, rather than caching that somewhere (say in the zfsvfs_t), decides to go through the whole zfs_zget lookup path. Since the object ID is the same, those calls all hash to the same lock, so the multiple CPUs (16 threads, in this case) have a tendency to enter heavy contention. Quite a bit of blocked-mutex time could probably be eliminated by caching that vnode for the life of the mount.

Actions #1

Updated by Electric Monk about 2 years ago

  • Gerrit CR set to 2040
Actions #2

Updated by Patrick Mooney about 2 years ago

I tested this atop omnios-r151040 bits on my build machine, when comparing a build of the same tree:

OLD:
  ==== Total build time ====

  real    0:36:53

  ==== Elapsed build time (non-DEBUG) ====

  real    17:58.0
  user  2:38:51.7
  sys     49:43.3

  ==== Elapsed build time (DEBUG) ====

  real     9:23.3
  user  1:32:13.7
  sys     27:08.8

NEW:

  ==== Total build time ====

  real    0:35:36

  ==== Elapsed build time (non-DEBUG) ====

  real    17:19.3
  user  2:36:14.4
  sys     46:08.2

  ==== Elapsed build time (DEBUG) ====

  real     8:58.0
  user  1:30:52.0
  sys     25:26.5
Actions #3

Updated by Robert Mustacchi about 2 years ago

  • Assignee set to Patrick Mooney
Actions #4

Updated by Patrick Mooney over 1 year ago

Some additional testing I did around mount/umount behavior:

On a system running the proposed change, I created a zfs volume and created a snapshot from it. I legacy mounted the volume, and attempted unmounts both when a process was in the mountpoint (thus holding the root open) as well as in the .zfs ctldir. Both attempts failed with EBUSY. When the obstructing process (a shell) was moved out of the directories in question, the umount proceeded normally. This was repeated with a traditional mount setup (mountpoint!=legacy) traversing into the normal volume and the ctldir, attempting umounts when they would normally be blocked (resulting in EBUSY) and confirming they function as desired when the blocker is removed. After these tests, the machine was force-dumped to check for memory leaks (none were found). This was also repeated with a forced (zfs unmount -f) unmount to confirm that it both works, and also does not result in a resource leak.

Actions #5

Updated by Patrick Mooney over 1 year ago

  • Status changed from New to Pending RTI
  • % Done changed from 0 to 90
Actions #6

Updated by Electric Monk over 1 year ago

  • Status changed from Pending RTI to Closed
  • % Done changed from 90 to 100

git commit 00bfaff92dad2fa278f0e40718333cf4864ad7d5

commit  00bfaff92dad2fa278f0e40718333cf4864ad7d5
Author: Patrick Mooney <pmooney@pfmooney.com>
Date:   2022-10-03T18:01:55.000Z

    14519 zfs_root should cache vnode
    Reviewed by: Dan McDonald <danmcd@mnx.io>
    Reviewed by: Toomas Soome <tsoome@me.com>
    Reviewed by: Andy Fiddaman <illumos@fiddaman.net>
    Approved by: Gordon Ross <gordon.w.ross@gmail.com>

Actions

Also available in: Atom PDF