Disabled MSI-X for NVMe passthrough on ESXi must be reverted because VMware fixed the bug on their end
Back in 2017 there was a VMware bug with NVMe passthrough that prevented the NVMe driver to load correctly. It was caused by MSI-X interactions between VMware and the driver.
A fix was provided in illumos-gate: https://www.illumos.org/issues/7723 It was a simple work around, to avoid using MSI-X altogether.
Meanwhile, VMware fixed the issue: https://kb.vmware.com/s/article/2150585
The side effect of the VMware fix is that the current NVMe driver in illumos-gate is not working any more on any of the updated VMware versions.
We reverted the MSI-X fix and confirmed on our OmniOSce build that after the revert the NVMe dirver loads correctly and the drives can be seen. We did not do any further testing, yet.
Updated by Csaba Patkos about 3 years ago
Yuri Pankov wrote:
This would require knowing the exact version of VMware hypervisor we are running on, and I'm not aware of any way of doing that from kernel space (i.e. very early in the boot).
Well, if that cannot be done, it is a choice between supporting old VMware or newer VMware. The update mentioned in linked vmware KB article points to ESX 6.0 Update 3 which was released in February 2017, one and a half years ago.
I guess it is safe to assume that most people are using a VMware version that contains the fix.