This started as an exercise to improve concurreny through blkdev (https://www.illumos.org/issues/11827 ). Whilst testing that I discovered that the single taskq used to handle command completions is also a bottleneck (https://www.illumos.org/issues/11847 ).
This fixes both and when using vdbench, shows significant performance benefits - again see https://www.illumos.org/issues/11827 for benchmark results.
The changes in blkdev are to provide multiple wait/runq per device.

Som of the changes in nvme are for the multiple blkdev queues, the majority are changing the single command completion taskq, to one taskq per completion queue.

I ran vdbench as a stress test and to confirm performance.