zfs_root should cache vnode
Partaking in my pastime of dtracing a node while it's performing an illumos-gate build, I found some
lockstat(1) data which seemed to warrant additional investigation. A top contender in the adaptive-mutex-blocked category was a lock being acquired in the
zfs_zget function. Further investigation lead me to the most common stack (accounting for time spent blocked on the lock):
zfs`zfs_zget+0x62 zfs`zfs_root+0x60 genunix`fsop_root+0x33 genunix`traverse+0x90 genunix`lookuppnvp+0x2d9 genunix`lookuppnatcred+0x134 genunix`lookuppn+0x2e genunix`dogetcwd+0x15f genunix`getcwd+0x9b unix`sys_syscall+0x1a8 8035755520
It seems that every time we traverse a mount point, we ask the underlying filesystem for its root vnode. ZFS, rather than caching that somewhere (say in the
zfsvfs_t), decides to go through the whole
zfs_zget lookup path. Since the object ID is the same, those calls all hash to the same lock, so the multiple CPUs (16 threads, in this case) have a tendency to enter heavy contention. Quite a bit of blocked-mutex time could probably be eliminated by caching that vnode for the life of the mount.