illumos gate

Upstream clone of OS-7314 from SmartOS:

The changes and testing required for OS-7300 have spurred a renewed focus on ZFS performance in the face of large async-write workloads featuring interspersed fsync() calls to flush pending data to the disks. In typical test scenarios, when the async writes were progressing, IO to disks underlying the pool (3 pairs of mirrored SSDs, no SLOG) would look good, each disk pushing ~100MB/s at latencies around 1ms. With a large amount of data still outstanding, the calls to fsync() would change the performance profile drastically. The disks still report 100% busy via iostat, but their IOPS and throughput would fall dramatically, with average latency shooting up to 50ms or more. When the workload was updated to omit the fsync() (or at least delay it until the cached writes had completed), throughput to the disks would remain high and the total write workload would complete in a timely fashion.

Pending code review and discussion at openzfs#703