zfs loading and unloading metaslabs at high frequency
This is Joyent bug OS-7151 (https://smartos.org/bugview/OS-7151) There is a lot of detail in that bug report. To summarize here, when we are experiencing this performance problem, there are some set of metaslabs that are being loaded and unloaded constantly. Not all systems will experience this problem and we were unable to recreate this scenario on a lab machine, despite trying for several months. However, when a system does experience this problem, what we observe using 'zdb -mm' is that the metaslab fragmentation profile is tightly clustered. That is, all of the metaslabs in the top-level vdevs have very similar fragmentation as shown by zfs. Machine which don't experience this problem have varying degrees of fragmentation across their metaslabs.