illumos gate

Partaking in my pastime of dtracing a node while it's performing an illumos-gate build, I found some lockstat(1) data which seemed to warrant additional investigation. A top contender in the adaptive-mutex-blocked category was a lock being acquired in the zfs_zget function. Further investigation lead me to the most common stack (accounting for time spent blocked on the lock):

              zfs`zfs_zget+0x62
              zfs`zfs_root+0x60
              genunix`fsop_root+0x33
              genunix`traverse+0x90
              genunix`lookuppnvp+0x2d9
              genunix`lookuppnatcred+0x134
              genunix`lookuppn+0x2e
              genunix`dogetcwd+0x15f
              genunix`getcwd+0x9b
              unix`sys_syscall+0x1a8
       8035755520

It seems that every time we traverse a mount point, we ask the underlying filesystem for its root vnode. ZFS, rather than caching that somewhere (say in the zfsvfs_t), decides to go through the whole zfs_zget lookup path. Since the object ID is the same, those calls all hash to the same lock, so the multiple CPUs (16 threads, in this case) have a tendency to enter heavy contention. Quite a bit of blocked-mutex time could probably be eliminated by caching that vnode for the life of the mount.