Project

General

Profile

Actions

Bug #14519

open

zfs_root should cache vnode

Added by Patrick Mooney 6 months ago. Updated 6 months ago.

Status:
New
Priority:
Normal
Category:
zfs - Zettabyte File System
Start date:
Due date:
% Done:

0%

Estimated time:
Difficulty:
Medium
Tags:
Gerrit CR:

Description

Partaking in my pastime of dtracing a node while it's performing an illumos-gate build, I found some lockstat(1) data which seemed to warrant additional investigation. A top contender in the adaptive-mutex-blocked category was a lock being acquired in the zfs_zget function. Further investigation lead me to the most common stack (accounting for time spent blocked on the lock):

              zfs`zfs_zget+0x62
              zfs`zfs_root+0x60
              genunix`fsop_root+0x33
              genunix`traverse+0x90
              genunix`lookuppnvp+0x2d9
              genunix`lookuppnatcred+0x134
              genunix`lookuppn+0x2e
              genunix`dogetcwd+0x15f
              genunix`getcwd+0x9b
              unix`sys_syscall+0x1a8
       8035755520

It seems that every time we traverse a mount point, we ask the underlying filesystem for its root vnode. ZFS, rather than caching that somewhere (say in the zfsvfs_t), decides to go through the whole zfs_zget lookup path. Since the object ID is the same, those calls all hash to the same lock, so the multiple CPUs (16 threads, in this case) have a tendency to enter heavy contention. Quite a bit of blocked-mutex time could probably be eliminated by caching that vnode for the life of the mount.

Actions #1

Updated by Electric Monk 6 months ago

  • Gerrit CR set to 2040
Actions #2

Updated by Patrick Mooney 6 months ago

I tested this atop omnios-r151040 bits on my build machine, when comparing a build of the same tree:

OLD:
  ==== Total build time ====

  real    0:36:53

  ==== Elapsed build time (non-DEBUG) ====

  real    17:58.0
  user  2:38:51.7
  sys     49:43.3

  ==== Elapsed build time (DEBUG) ====

  real     9:23.3
  user  1:32:13.7
  sys     27:08.8

NEW:

  ==== Total build time ====

  real    0:35:36

  ==== Elapsed build time (non-DEBUG) ====

  real    17:19.3
  user  2:36:14.4
  sys     46:08.2

  ==== Elapsed build time (DEBUG) ====

  real     8:58.0
  user  1:30:52.0
  sys     25:26.5
Actions #3

Updated by Robert Mustacchi 6 months ago

  • Assignee set to Patrick Mooney
Actions

Also available in: Atom PDF