Feature #14081


bhyve upstream sync 2021 September

Added by Andy Fiddaman 4 months ago. Updated 2 months ago.

Start date:
Due date:
% Done:


Estimated time:
Gerrit CR:


Sync upstream activity in FreeBSD's bhyve into illumos.

The following list shows the commits brought over as part of this sync.
The character in the first column indicates the action taken:

* - patch taken
S - patch skipped
A - patch already in illumos
*    1 bhyve: Enable virtio-scsi legacy config parsing.
*    2 bhyve: change vq_getchain to return iovecs in both directions
*    3 AMD-vi: Fix mismatched NULL checking in amdiommu teardown path
*    4 AMD-vi: Mixed format IVHD block should replace fixed format IVHD block
A    5 Fix typo in xhci nvlist node name, and also increment device counter.
A    6 bhyve: fix regression in legacy virtio-9p config parsing
*    7 bhyve: add SMBIOS Baseboard Information
*    8 bhyve: Move the gdb_active check to gdb_cpu_suspend().
*    9 bhyve: implement rdmsr for MSR_IA32_FEATURE_CONTROL
        (already handled in the illumos kernel, but also add to userland to keep sync with FreeBSD)
*   10 bhyve.8: Make synopsis more readable
*   11 bhyve: Fix synopsis in the usage message
*   12 bhyve: Improve the option description in the usage message
*   13 bhyve.8: Sort the options in the OPTIONS section
*   14 bhyve.8: Improve the description and synopsis of -l
*   15 bhyve.8: Improve the description of the -m flag
S   16 bhyve.8: Fix the synopsis of -p
S   17 bhyve.8: Clean up description of -r              (snapshot)
*   18 bhyve.8: Fix indention in the signals table
*   19 bhyve.8: Clean-up synopsis of -s
*   20 bhyve.8: Clean up the slot description of -s
*   21 bhyve.8: Improve emulation description of the -s flag
S   22 bhyve.8: Clean up network backends section
*   23 bhyve.8: Clean up block storage device backends description
S   24 bhyve.8: Clean up SCSI device backends section
*   25 bhyve.8: Clean up 9P device backends section
*   26 bhyve.8: Clean up TTY, boot ROM, and pass-through descriptions
*   27 bhyve.8: Clean up virtio console device backends description
*   28 bhyve.8: Improve framebuffer backends description
*   29 bhyve.8: Improve documentation of NVME backend
*   30 bhyve.8: Improve AHCI backends documentation
S   31 bhyve: Document the format for HD audio backends
*   32 bhyve.8: Fix mandoc -Tlint issues
*   33 AMD-vi: Fortify IVHD device_identify process
*   34 bhyve: Be explicit that setting config.dump will not start a VM.
*   35 Append Keyboard Layout specified option for using VNC.
*   36 bhyve: Gracefully handle virtio-scsi with no conf
*   37 bhyve: Set SO_REUSEADDR on the gdb stub socket
*   38 libvmm: explicitly save and restore errno in vm_open()
S   39 bhyve/snapshot: provide a way to send other messages/data to bhyve
S   40 bhyve/snapshot: split up mutex/cond initialization from socket creation
*   41 vmm: Let guests enable SMEP/SMAP if the host supports it
*   42 Add a virtio-input device emulation.             (but not wired up)
*   43 bhyve: Register new kevents synchronously.
*   44 bhyve: Add support for EVFILT_VNODE mevents.
*   45 bhyve: Add support for handling disk resize events to block_if.
*   46 bhyve: Split out a lower-level helper for VirtIO interrupts.
*   47 bhyve vtblk: Inform guests of disk resize events.
*   48 bhyve: enhance debug info for memory range clash
*   49 bhyve: Fix cli regression with NVMe ram          (replaces 13860)
*   50 vmm: Fix ivrs_drv device_printf usage
*   51 bhyve: fix NVMe MDTS comment
*   52 bhyve: Fix NVMe iovec construction for large IOs
*   53 vmm: Fix AMD-vi using wrong rid range
*   54 libvmmapi: Fix warnings and stop overridding WARNS
*   55 bhyve: net_backends, automatically IFF_UP tap devices
S   56 vmm: Bump vmname buffer in struct vm to VM_MAX_NAMELEN + 1
*   57 bhyve: Use fspacectl(2) for BOP_DELETE on regular file images
A   58 amd64: Fix output operand specs for the stmxcsr and vmread intrinsics
*   59 vmm: Make iommu ops tables const
*   60 vmm: Fix wrong assert in ivhd_dev_add_entry
S   61 bhyve: Use pci(4) to access I/O port BARs
S   62 vmm: Add credential to cdev object
*   63 byhve: add option to specify IP address for gdb
*   64 bhyve: change a default address from ANY to localhost
*   65 bhyve: Fix vq_getchain() error handling bugs in various device models
*   66 bhyve: Nuke double-semicolons
S   67 Create sys/reg.h for the common code previously in machine/reg.h
*   68 bhyve: Fix pci device node key in bhyve_config.5
A   69 Remove an always-true check.

Related issues

Related to illumos gate - Bug #14100: bhyve misses TLB flush for shadowed cr0ClosedPatrick Mooney

Blocked by illumos gate - Feature #13380: Add virtio-9p (aka VirtFS) filesystem sharing to bhyveClosedAndy Fiddaman

Actions #1

Updated by Andy Fiddaman 4 months ago

  • Blocked by Feature #13380: Add virtio-9p (aka VirtFS) filesystem sharing to bhyve added
Actions #2

Updated by Andy Fiddaman 4 months ago

  • Description updated (diff)
Actions #3

Updated by Andy Fiddaman 4 months ago

  • Description updated (diff)
Actions #4

Updated by Andy Fiddaman 4 months ago

Testing the new feature to report a size change for a vioblk disk to the guest.
This is using a raw file rather than a zvol for the backing store since sdev does not support vnode event notifications, but it proves the principle and the changes to mevent.c

# ls -l /data/af/iso/testdisk
-rw-r--r--   1 root     root          2G Sep 16 14:05 /data/af/iso/testdisk
# truncate -s 4G /data/af/iso/testdisk
# ls -l /data/af/iso/testdisk
-rw-r--r--   1 root     root          4G Sep 16 14:05 /data/af/iso/testdisk

and chmod to cause an update to ctime which will raise the event

# ls -% all testdisk
         timestamp: atime         Sep 16 14:06:42 2021
         timestamp: ctime         Sep 16 14:06:52 2021
         timestamp: mtime         Sep 16 14:06:52 2021
         timestamp: crtime         Sep 16 13:49:03 2021
nemesis# chmod 640 testdisk
nemesis# ls -% all testdisk
         timestamp: atime         Sep 16 14:06:42 2021
         timestamp: ctime         Sep 16 14:07:12 2021
         timestamp: mtime         Sep 16 14:07:12 2021
         timestamp: crtime         Sep 16 13:49:03 2021

and then, in the guest (Debian):

[  305.739600] virtio_blk virtio0: [vda] new size: 8388608 512-byte logical blocks (4.29 GB/4.00 GiB)
[  305.741970] vda: detected capacity change from 2147483648 to 4294967296
Actions #5

Updated by Andy Fiddaman 4 months ago

  • Category set to bhyve
  • Gerrit CR set to 1709
Actions #6

Updated by Jorge Schrauwen 4 months ago

Been testing the bits with the patch archive provided by andy and my win10 VM is hitting:

vm exit[1]
    reason        VMX
    rip        0x00000000000016d7
    inst_length    3
    status        0
    exit_reason    2 (Triple fault)
    qualification    0x0000000000000000
    inst_type        0
    inst_error        0
fbuf frame buffer base: fffffc7fed200000 [sz 16777216]

(from /tmp/init.log)

It hits this ~1-2 sec after boot just when the tianocore logo shows up on VNC. My freebsd and linux VM's are fine but I don't use VNC on those.

I need this VM so will reverst to the old BE now, but keep it around if more testing is needed.


# Generated from zone configuration
smbios.product=OmniOS HVM
smbios.sku=001 Machine

Actions #7

Updated by Andy Fiddaman 4 months ago

Using the old bhyve binary along with the updated kernel produces the same result, so this is likely related to a change in the kernel. The best candidate is * 41 vmm: Let guests enable SMEP/SMAP if the host supports it

Reverting this by hot-patching the kernel:

407:#define   CPUID_STDEXT_SMEP  0x00000080
420:#define   CPUID_STDEXT_SMAP  0x00100000
> x86_emulate_cpuid+0xf2::dis -n0
x86_emulate_cpuid+0xf2:               andl   $0x3c150bb9,0xffffffffffffff94(%rbp)
> x86_emulate_cpuid+0xf4/X
x86_emulate_cpuid+0xf4:         150bb994
> x86_emulate_cpuid+0xf4/W050b3994
x86_emulate_cpuid+0xf4:         0x150bb994      =       0x50b3994
> x86_emulate_cpuid+0xf2::dis -n0
x86_emulate_cpuid+0xf2:         andl   $0x3c050b39,0xffffffffffffff94(%rbp)

and now a test windows vm boots successfully.

Actions #8

Updated by Patrick Mooney 4 months ago

  • Related to Bug #14100: bhyve misses TLB flush for shadowed cr0 added
Actions #9

Updated by Andy Fiddaman 4 months ago

The problem mentioned above has been resolved by the integration of #14100

Actions #10

Updated by Andy Fiddaman 2 months ago

This has been in OmniOS bloody for a couple of months and tested with a wide range of guests including windows 10, windows 11, freebsd, various linux distributions, openindiana, solaris and omnios.
Guest configurations included variations in framebuffer, pass-through, emulated disk interface (ahci, nvme, vioblk) etc.
Thanks for Jorge and Mike for additional testing here.

I verified that guests now see the SMEP and SMAP feature bits passed through.

Actions #11

Updated by Electric Monk 2 months ago

  • Status changed from In Progress to Closed
  • % Done changed from 0 to 100

git commit b0de25cb23668fa4535078d18a0618eee442c000

commit  b0de25cb23668fa4535078d18a0618eee442c000
Author: Andy Fiddaman <>
Date:   2021-11-23T09:04:03.000Z

    14081 bhyve upstream sync 2021 September
    Reviewed by: Patrick Mooney <>
    Reviewed by: Hans Rosenfeld <>
    Approved by: Robert Mustacchi <>


Also available in: Atom PDF