illumos gate

The pvscsi driver used with VMware was unfortunately designed to some legacy APIs, and we have observed some issues with it.

• It's support for hot plug support is "limited" -- we haven't observed that it actually works at all despite the code present (and it has not support for removal)
• It can run out of resources (contexts) and issues warnings on the console when this occurs.
• In reset circumstances, it can crash due to a circular lock associated with the completion ring handling

Additionally:

1. The driver should be updated to support modern SCSA DMA interfaces (tran_setup_pkt() and friends)
2. The driver should use the iport(9) interfaces for creating target ports
3. In theory the driver could benefit from LUN notifications (though we don't know if any hypervisors support this, the interface allows for non-zero lun notifications)
4. It should be moved out of intel/ - because VMWare runs on Aarch64, and aarch64 support is anticipated to be added to illumos (some folks have made it work already)
5. It should use correct DMA synchronization methods instead of explicit memory barriers (which are actually more expensive and normally not needed)
6. The DMA attributes need to be corrected -- apart from page alignment considerations some of the other restrictions (e.g. to the lower 32-bits of memory) are actually incorrect.
7. The support for quiescing the SCSI bus is of zero value, and adds a lot of extra complexity -- modern SAS and virtual busses do not need to quiesce SCSI busses (quiesce was created to allow legacy parallel busses to be electrically quiesced so that they could be made safe for configuration changes -- modern busses are hot-plug safe)
8. The attempt to use separate locks for the completion ring and submission rings is misguided -- likely based on thinking from NIC drivers. HBAs move a lot more data per operation, and spend a lot less time holding these locks when interacting with these paths, and the simplicity from a single ring brings clear sustainability benefits without any measurable impact on performance.

When all these changes are made, the pvscsi can be reduced to about 60% of it's original size, with no loss in functionality, and several bugs are fixed along the way. This work can be easily done in a way that the pvscsi driver becomes 100% ddi compliant.