illumos gate

sjorge tested this atop SmartOS:

Applied this on top of SmartOS PI I build earlier today. Tested the following OS's and all still good
- FreeBSD
- NetBSD
- OpenBSD
- OmniOS
- Windows 10

FreeBSD and NetBSD was done on:
Supermicro X9DRi-LN4+/X9DR3-LN4+ + 2x Intel(R) Xeon(R) CPU E5-2620 v2 @ 2.10GHz (Ivy Bridge EP, so It should have APICv)

FreeBSD, OpenBSD, OmniOS, and Windows:
Supermicro SSG-5029P-E1CTR12L + 1x Intel(R) Xeon(R) Silver 4110 CPU @ 2.10GHz (Skylake EP (IIRC))

I applied this patchset on top of illumos-joyent and ran some tests.

Ran various linux guests such as Ubuntu (20.04/18.04), and Archlinux (kernel 5.6.15-arch1-1).

Physical Hardware:
Intel NUC (Version String: KYSKLi70.86A.0051.2018.0110.1800)
Intel(r) Core(tm) i7-6770HQ CPU @ 2.60GHz

Linux KVM with nested virt:
AMD Ryzen Threadripper 1950X 16-Core Processor

I also confirmed that I could run through the Ubuntu 20.04 installer multiple times without seeing the segfault umouse_request(), which is present without the resync. There's also an upstream patch that hardens this code even though the MSI patch included in the resync seems to fix the issue.

I verified that the bhyve virtual nvme device still works as there were a bunch of changes in that code. Finally I tested passing through a physical nvme drive into a guest.

[root@deepnest ~]# nvme list
Node             SN                   Model                                    Namespace Usage                      Format           FW Rev
---------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- --------
/dev/nvme0n1     NVME-0-5             bhyve-NVMe                               1           5.37  GB /   5.37  GB    512   B +  0 B   1.0
/dev/nvme1n1     BTPY72940FJ2256D     INTEL SSDPEKKF256G7L                     1         256.06  GB / 256.06  GB    512   B +  0 B   121P

[root@deepnest ~]# lsblk -f
NAME        FSTYPE FSVER LABEL UUID                                 FSAVAIL FSUSE% MOUNTPOINT
vda
├─vda1      vfat   FAT32       2084-5FD4
└─vda2      ext4   1.0         8cdf7f8e-84a1-41af-b6be-3c7523d8eff7    3.5G    76% /
vdb         xfs                8b0387bd-299f-4823-9190-a4ce884674a3
nvme0n1     xfs                8b0387bd-299f-4823-9190-a4ce884674a3
nvme1n1
└─nvme1n1p1 ext4   1.0         280d8aaf-b635-4c38-8c4b-46d20edc8e7e
[root@deepnest ~]# mount -t xfs /dev/nvme0n1 /mnt/a
[root@deepnest ~]# mount -t ext4 /dev/nvme1n1p1 /mnt/b
[root@deepnest ~]# cd /mnt/a
[root@deepnest a]# ls
foo  testfile
[root@deepnest a]# touch sync
[root@deepnest a]# ls
foo  sync  testfile
[root@deepnest a]# cd /mnt/b
[root@deepnest b]# ls
lost+found  test  testfile
[root@deepnest b]# touch sync
[root@deepnest b]# ls
lost+found  sync  test    testfile

After a small fix, the mevent tests are happy again:

Executing test /opt/bhyvetest/tst/mevent/lists.delete.exe ... passed
Executing test /opt/bhyvetest/tst/mevent/read.disable.exe ... passed
Executing test /opt/bhyvetest/tst/mevent/read.pause.exe ... passed
Executing test /opt/bhyvetest/tst/mevent/read.requeue.exe ... passed

-------------
Results
-------------

Tests passed: 4
Tests failed: 0
Tests ran:    4

Both sjorge and Mike Zeller re-ran their tests after the mevent update and found them behaving properly as expected.