bhyve upstream sync 2022 January
Sync upstream activity in FreeBSD's bhyve into illumos.
The following list shows the commits brought over as part of this sync.
The characters in the first columns indicate the action taken:
S - skipped U - updated to match upstream ~ - partially taken * - merged F - only freebsd-specific code changed A - already had
UA 1 bhyve: Support setting the disk serial number for VirtIO block devices. S 2 cpuset(9): Add CPU_FOREACH_IS(SET|CLR) and modify consumers to use it S 3 Revert "cpuset(9): Add CPU_FOREACH_IS(SET|CLR) and modify consumers... SA 4 bhyve: Update the -G description in the SYNPOSIS. * 5 bhyve_config.5: Document gdb.address. *F 6 bhyve: Add an empty case for event types in mevent_kq_fflags(). * 7 bhyve: Map the MSI-X table unconditionally for passthrough * 8 bhyve.8: Fix markup of the -G flag * 9 bhyve: Update usage and synopsis for the -k flag * 10 bhyve: ignore low bits of CFGADR S 11 cpuset(9): Add CPU_FOREACH_IS(SET|CLR) and modify consumers to use it S 12 bhyve: Fix the WITH_BHYVE_SNAPSHOT build ~ 13 vlapic: Schedule callouts on the local CPU Created vlapic_callout_reset() and used it, but not moving the cyclic since it's already localised via cyclic_move_here() early on * 14 bhyve: Bump the SMBIOS firmware version to 14.0 for 14-CURRENT * 15 bhyve: use physical lobits for BARs of passthru devices * 16 bhyve: do not explicitly map fbuf framebuffer * 17 bhyve: move 64 bit BAR location to match OVMF assumptions * 18 bhyve: use a fixed 32 bit BAR base address * 19 bhyve: keep physical and virtual COMMAND reg in sync * 20 bhyve: emulate reads of MSI-X capabilities for passthru devices * 21 bhyve: Fix compile * 22 bhyve blockif: fix blockif_candelete with Capsicum * 23 bhyve: fix -Wunused-but-set-variable warning * 24 bhyve: Support a _VARS.fd file for bootrom *F 25 bhyve: set EV_CLEAR for EVFILT_VNODE mevents * 26 bhyve nvme: Add AEN support to NVMe emulation * 27 bhyve nvme: Inform guests of namespace resize S 28 bhyve: Only snapshot initialized VirtIO queues * 29 bhyve: passthru: enable BARs before possibly mmap(2)ing them * 30 bhyve: clean up trailing whitespaces SA 31 bhyve smbios type 3 structure is incorrect S 32 Make CPU_SET macros compliant with other implementations * 33 bhyve: only init MSI-X table if passthru device supports it * 34 bhyve: enumerate BARs by size * 35 bhyve: allow reading of fwctl signature multiple times * 36 bhyve: add more slop to 64 bit BARs * 37 bhyve: dynamically register FwCtl ports
Of note in this wad are:
- Support for persistent UEFI variables;
- AEN support in NVMe, with notifications being posted if the device size changes;
- Improvements in passthrough, including workarounds for devices or guest drivers which do not adhere to the PCI spec.
Updated by Andy Fiddaman 4 months ago
This has undergone the usual gamut of tests, running a variety of guests including Windows 10/11, FreeBSD, Linux (Ubuntu) and OmniOS.
Early tests with the new persistent UEFI variable support exposed #14367 which Patrick Mooney has since resolved (and added tests for).
I tested the new persistent UEFI variables using the UEFI shell - setting them, destroying the VM and then checking that they were still there after booting it back up.
Jorge Schrauwen has run it on two of his servers which use passthrough of some network devices and ports on a quad xhci card and was instrumental in early testing and feedback, helping to resolve passthrough issues from this change - in particular #14373 which is now fixed in illumos-gate.
I've done extensive testing of the new NVMe AEN support and was able to confirm that the event propagates to the guest and that the guest is able to read the changed namespaces log page and see that namespace 1 (the only one supported by bhyve's NVMe emulation). This showed up a small bug in that the virtual NVMe device was not setting the correct bit in the Optional Asynchronous Event Support (OAES) field to indicate support for these namespace change AENs and the corresponding log page. This is fixed in this wad and will be reported back/upstreamed to FreeBSD. Note that this feature requires #14374 to work with disks backed by a ZVOL.
This test used a patched
nvmeadm binary and
nvme driver to expose access to the log page. #14467 updates the driver so that it displays more useful messages (but still does not act on the notification other than to clear the log page)
root@bhyvetest:~# nvmeadm list nvme0: model: bhyve-NVMe, serial: NVME-4-0, FW rev: 1.0, NVMe v1.3 nvme0/589cfc2045c20001 (c3t589CFC2045C20001d0): Size = 31744 MB, Capacity = 31744 MB, Used = 31744 MB root@bhyvetest:~# dmesg | grep nvme Feb 1 16:03:44 bhyvetest nvme: [ID 339590 kern.notice] NOTICE: nvme0: one or more namespace attributes changed, logpage = 4 root@bhyvetest:~# nvmeadm get-logpage nvme0 nschange nvme0: Changed namespace list: Namespace : 1
Updated by Electric Monk 4 months ago
- Status changed from In Progress to Closed
- % Done changed from 0 to 100
commit 6dc983494b0ffef2565cc4d91371ee345425ffab Author: Andy Fiddaman <firstname.lastname@example.org> Date: 2022-02-07T20:51:32.000Z 14372 bhyve upstream sync 2022 January Reviewed by: Patrick Mooney <email@example.com> Reviewed by: Toomas Soome <firstname.lastname@example.org> Reviewed by: Jorge Schrauwen <email@example.com> Approved by: Dan McDonald <firstname.lastname@example.org>