illumos gate

Upstreaming OS-8143 from SmartOS:

This is a follow up to OS-8117. This pulls in an initially skipped userland diff for pci passthru.
The diff comes from FreeBSD commit 8e4b03df46ab (r348779).

Additionally we want to grab https://github.com/freebsd/freebsd/commit/bdad744823ebf970815d691c9ec71236f6f0d90b ahead of the next re-sync as the commit is directly related to r348779.

To test this change I booted a debug platform with the patch on an intel NUC.

I setup loader.conf.local to look like:

[root@hollownest ~]# cat /opt/local/bin/mount-usb^C
[root@hollownest ~]# cat /usb
usbkey/   usbstick/
[root@hollownest ~]# cat /usbstick/boot/loader.conf.local
ppt_aliases_load=YES
ppt_aliases_type=file
ppt_aliases_name=/overlay/etc/ppt_aliases
ppt_aliases_flags="name=/etc/ppt_aliases" 

ppt_matches_load=YES
ppt_matches_type=file
ppt_matches_name=/overlay/etc/ppt_matches
ppt_matches_flags="name=/etc/ppt_matches" 

With each file looking like:

[root@hollownest ~]# cat /usbstick/overlay/etc/ppt_matches
pci8086,390a
[root@hollownest ~]# cat /usbstick/overlay/etc/ppt_aliases
ppt "/pci@0,0/pci8086,a118@1d/pci8086,390a@0" 

This sets aside the nvme device in the NUC for use with a bhyve VM.

[root@hollownest ~]# pptadm list -a -o dev,vendor,device,path
DEV        VENDOR DEVICE PATH
/dev/ppt0  8086   f1a5   /pci@0,0/pci8086,a118@1d/pci8086,390a@0

Using the above device I updated the VM with the following json payload:

{
  "pci_devices": [
    {
      "path": "/devices/pci@0,0/pci8086,a118@1d/pci8086,390a@0",
      "pci_slot": "0:8:0" 
    }
  ]
}

Inside the VM I provisioned an ext4 filesystem and wrote some data to a file.

[root@deepnest ~]# lsblk /dev/nvme1n1 -f
NAME FSTYPE FSVER LABEL UUID                                 FSAVAIL FSUSE% MOUNTPOINT
nvme1n1
│
└─nvme1n1p1
     ext4   1.0         280d8aaf-b635-4c38-8c4b-46d20edc8e7e  221.8G     0% /ppt
[root@deepnest ~]# echo hello > /ppt/testfile
[root@deepnest ~]# cat /ppt/testfile
hello

Additionally I had sjorge in the community test the patch on his hardware. He was able to passthrough the previously problematic NIC with multiple BARs to his Windows VM and everything worked as expected.