Project

General

Profile

Actions

Feature #14372

closed

bhyve upstream sync 2022 January

Added by Andy Fiddaman 5 months ago. Updated 4 months ago.

Status:
Closed
Priority:
Normal
Assignee:
Category:
bhyve
Start date:
Due date:
% Done:

100%

Estimated time:
Difficulty:
Medium
Tags:
Gerrit CR:

Description

Sync upstream activity in FreeBSD's bhyve into illumos.

The following list shows the commits brought over as part of this sync.
The characters in the first columns indicate the action taken:

S - skipped
U - updated to match upstream
~ - partially taken
* - merged
 F - only freebsd-specific code changed
 A - already had

UA   1  bhyve: Support setting the disk serial number for VirtIO block devices.
S    2  cpuset(9): Add CPU_FOREACH_IS(SET|CLR) and modify consumers to use it
S    3  Revert "cpuset(9): Add CPU_FOREACH_IS(SET|CLR) and modify consumers...
SA   4  bhyve: Update the -G description in the SYNPOSIS.
*    5  bhyve_config.5: Document gdb.address.
*F   6  bhyve: Add an empty case for event types in mevent_kq_fflags().
*    7  bhyve: Map the MSI-X table unconditionally for passthrough
*    8  bhyve.8: Fix markup of the -G flag
*    9  bhyve: Update usage and synopsis for the -k flag
*   10  bhyve: ignore low bits of CFGADR
S   11  cpuset(9): Add CPU_FOREACH_IS(SET|CLR) and modify consumers to use it
S   12  bhyve: Fix the WITH_BHYVE_SNAPSHOT build
~   13  vlapic: Schedule callouts on the local CPU
                Created vlapic_callout_reset() and used it, but not moving
                the cyclic since it's already localised via cyclic_move_here()
                early on
*   14  bhyve: Bump the SMBIOS firmware version to 14.0 for 14-CURRENT
*   15  bhyve: use physical lobits for BARs of passthru devices
*   16  bhyve: do not explicitly map fbuf framebuffer
*   17  bhyve: move 64 bit BAR location to match OVMF assumptions
*   18  bhyve: use a fixed 32 bit BAR base address
*   19  bhyve: keep physical and virtual COMMAND reg in sync
*   20  bhyve: emulate reads of MSI-X capabilities for passthru devices
*   21  bhyve: Fix compile
*   22  bhyve blockif: fix blockif_candelete with Capsicum
*   23  bhyve: fix -Wunused-but-set-variable warning
*   24  bhyve: Support a _VARS.fd file for bootrom
*F  25  bhyve: set EV_CLEAR for EVFILT_VNODE mevents
*   26  bhyve nvme: Add AEN support to NVMe emulation
*   27  bhyve nvme: Inform guests of namespace resize
S   28  bhyve: Only snapshot initialized VirtIO queues
*   29  bhyve: passthru: enable BARs before possibly mmap(2)ing them
*   30  bhyve: clean up trailing whitespaces
SA  31  bhyve smbios type 3 structure is incorrect
S   32  Make CPU_SET macros compliant with other implementations
*   33  bhyve: only init MSI-X table if passthru device supports it
*   34  bhyve: enumerate BARs by size
*   35  bhyve: allow reading of fwctl signature multiple times
*   36  bhyve: add more slop to 64 bit BARs
*   37  bhyve: dynamically register FwCtl ports

Of note in this wad are:

  • Support for persistent UEFI variables;
  • AEN support in NVMe, with notifications being posted if the device size changes;
  • Improvements in passthrough, including workarounds for devices or guest drivers which do not adhere to the PCI spec.

Related issues

Related to illumos gate - Feature #14469: nvme could raise dynamic lun expansion syseventsClosedAndy Fiddaman

Actions
Related to illumos gate - Bug #14536: bhyve: viona uses wrong address for queue notify ioportClosedAndy Fiddaman

Actions
Blocked by illumos gate - Bug #14367: bhyve gpt mishandles small mappingsClosedPatrick Mooney

Actions
Blocked by illumos gate - Bug #14373: ppt driver maps BAR to wrong ddi registerClosedAndy Fiddaman

Actions
Actions #1

Updated by Andy Fiddaman 5 months ago

  • Blocked by Bug #14367: bhyve gpt mishandles small mappings added
Actions #2

Updated by Andy Fiddaman 5 months ago

  • Blocked by Bug #14373: ppt driver maps BAR to wrong ddi register added
Actions #3

Updated by Andy Fiddaman 5 months ago

  • Description updated (diff)
Actions #4

Updated by Electric Monk 5 months ago

  • Gerrit CR set to 1934
Actions #5

Updated by Andy Fiddaman 4 months ago

This has undergone the usual gamut of tests, running a variety of guests including Windows 10/11, FreeBSD, Linux (Ubuntu) and OmniOS.

Early tests with the new persistent UEFI variable support exposed #14367 which Patrick Mooney has since resolved (and added tests for).
I tested the new persistent UEFI variables using the UEFI shell - setting them, destroying the VM and then checking that they were still there after booting it back up.

Jorge Schrauwen has run it on two of his servers which use passthrough of some network devices and ports on a quad xhci card and was instrumental in early testing and feedback, helping to resolve passthrough issues from this change - in particular #14373 which is now fixed in illumos-gate.

I've done extensive testing of the new NVMe AEN support and was able to confirm that the event propagates to the guest and that the guest is able to read the changed namespaces log page and see that namespace 1 (the only one supported by bhyve's NVMe emulation). This showed up a small bug in that the virtual NVMe device was not setting the correct bit in the Optional Asynchronous Event Support (OAES) field to indicate support for these namespace change AENs and the corresponding log page. This is fixed in this wad and will be reported back/upstreamed to FreeBSD. Note that this feature requires #14374 to work with disks backed by a ZVOL.

This test used a patched nvmeadm binary and nvme driver to expose access to the log page. #14467 updates the driver so that it displays more useful messages (but still does not act on the notification other than to clear the log page)

root@bhyvetest:~# nvmeadm list
nvme0: model: bhyve-NVMe, serial: NVME-4-0, FW rev: 1.0, NVMe v1.3
  nvme0/589cfc2045c20001 (c3t589CFC2045C20001d0): Size = 31744 MB, Capacity = 31744 MB, Used = 31744 MB

root@bhyvetest:~# dmesg | grep nvme
Feb  1 16:03:44 bhyvetest nvme: [ID 339590 kern.notice] NOTICE: nvme0: one or more namespace attributes changed, logpage = 4

root@bhyvetest:~# nvmeadm get-logpage nvme0 nschange
nvme0: Changed namespace list:
  Namespace :                               1
Actions #6

Updated by Andy Fiddaman 4 months ago

  • Related to Feature #14469: nvme could raise dynamic lun expansion sysevents added
Actions #7

Updated by Electric Monk 4 months ago

  • Status changed from In Progress to Closed
  • % Done changed from 0 to 100

git commit 6dc983494b0ffef2565cc4d91371ee345425ffab

commit  6dc983494b0ffef2565cc4d91371ee345425ffab
Author: Andy Fiddaman <omnios@citrus-it.co.uk>
Date:   2022-02-07T20:51:32.000Z

    14372 bhyve upstream sync 2022 January
    Reviewed by: Patrick Mooney <pmooney@pfmooney.com>
    Reviewed by: Toomas Soome <tsoome@me.com>
    Reviewed by: Jorge Schrauwen <sjorge@blackdot.be>
    Approved by: Dan McDonald <danmcd@joyent.com>

Actions #8

Updated by Andy Fiddaman 3 months ago

  • Related to Bug #14536: bhyve: viona uses wrong address for queue notify ioport added
Actions

Also available in: Atom PDF