Feature #12665


want modern bhyve

Added by Patrick Mooney about 2 years ago. Updated about 2 years ago.

Start date:
Due date:
% Done:


Estimated time:
Gerrit CR:


Once #12612 has landed, the next step is to import the series of commits (squash into one) which bring bhyve up for that initial port to the modern (FreeBSD 11) era. This will not be done piecemeal, since it would involve many cases of bugs introduced and then later fixed. We do not need to put the gate through those same pains.

Here's a listing of some of the illumos-joyent issues covered, although some of them are split up for later integration:
- OS-8008 bhyve on AMD should report SVM as disabled
- OS-7992 use native linker sets for bhyve build
- OS-7960 need fixes for NULL as a pointer
- OS-7790 enable smatch build for illumos-joyent (fix gcc7 build)
- OS-7447 formalize bhyve resource exclusion
- OS-7622 bhyve vioapic writes can deadlock instance
- OS-7819 bhyve upstream sync 2019 June
- OS-7835 bhyve modules should have mapfiles
- OS-7777 simplify bhyve vlapic ESR logic
- OS-7801 bhyve should expose MD_CLEAR capability
- OS-7768 illumos-joyent needs smatch fixes
- OS-7696 bhyve should avoid all-cpu cross calls
- OS-7170 bhyve should support AMD
- OS-7613 bhyve virtio needs barriers
- OS-7614 viona should use proper barriers
- OS-7580 bhyve upstream sync 2019 Feb
- OS-7583 bhyve goes awry on large memory systems
- OS-7438 refactor bhyve EPT to use generic page tables
- OS-7437 want generic indexed page table system
- OS-7528 update bhyve pkg manifest from OmniOS
- OS-7572 bhyve: stop shipping and
- OS-7424 bhyve: keep a list of deferred-destroyed VMs
- OS-7354 bhyve should avoid reserved PIR descriptor bits
- OS-7392 bhyve optstr wrong for !-Y
- OS-7446 clean up bhyve locking and names
- OS-7395 bhyve: VM_{SUSPEND,RESUME}_CPU ioctls ignore vcpu argument
- OS-7394 defer bhyve instance destruction to last close
- OS-7410 vmm_zsd assert tripped on zone shutdown
- OS-7397 reduce lock contention in bhyve instr emul
- OS-7420 bhyve needs fix for FreeBSD-SA-18:14.bhyve
- OS-7300 bhyve should not sync IO unless necessary
- OS-7416 bhyve BLOCKIF_NUMTHR should keep pace with MAXREQ
- OS-7369 full UART FIFO can send bhyve into a spin
- OS-7381 bhyve should handle UART socket reconnect
- OS-7238 bhyve zones should add hostbridge device
- OS-7265 bhyve should build with newer gcc
- OS-7012 bhyve wedged in vlapic cyclics
- OS-7016 vmx ctx ops should inform VMRESUME
- OS-6957 clean up unused mutex type from bhyve
- OS-7215 bhyve uart broken for win2016 guest
- OS-7175 BLOCKIF_IOV_MAX inadequate for Win2016 on bhyve
- OS-7078 NMI while in bhyve guest shouldn't panic()
- OS-7080 bhyve and KVM should coexist in peace
- OS-7125 Need mitigation of L1TF (CVE-2018-3646)
- OS-6620 bhyve reboot should reuse existing process
- OS-6966 SmartOS hangs in boot on bhyve due to recursive mutex in pci_virtio_block.c
- OS-7090 GDT limit reset to 0xffff
- OS-7054 bhyve should halt on triple fault
- OS-7061 bhyve: set MAP_NORESERVE when mmap'ing guest memory
- OS-7062 bhyve: return EINVAL for allocations > max_page_get
- OS-6832 reclaim memory when NOSLEEP bhyve allocation fails
- OS-6888 bhyve wedged on vioapic write
- OS-6923 bhyve could be more precise with vPIT
- OS-6849 bhyve should localize vatpit resources
- OS-7000 Need Eager FPU
- OS-6992 Want hypervisor API for FPU management
- OS-6999 bhyve should use HMA FPU framework
- OS-6960 bhyve TSC offsetting should be platform-agnostic
- OS-6988 bhyve wastes 1GB of RAM for each VM using more than 3GB
- OS-6912 bhyve virtio-blk should sync when needed
- OS-6793 vmm malloc/free are inefficient
- OS-6930 bhyve PIR can still miss PIR wake-ups
- OS-6976 vmm_zsd_lock not held while element removed from vmm_zsd_list
- OS-6954 bhyve upstream sync
- OS-6630 bhyve should support vnc
- OS-6907 vmm_stat initialization not run, causing 0-length allocation
- OS-6928 recursive mutex in pci_xhci
- OS-6869 sync bhyve with upstream
- OS-6864 bhyve should guard against going off-cpu
- OS-6865 bhyve could be lazy about FPU state
- OS-6858 vmm tracing framework is dead code
- OS-6861 bhyve should flush register state on vmx exit
- OS-6606 want memory arena for vmm applications
- OS-6835 memory DR should be disabled
- OS-6851 some bhyve atomics shims are too heavy-handed
- OS-6860 bhyve requires APIC cyclic be localized
- OS-6843 bhyve localize should respect cpu_lock
- OS-6837 bhyve must use separate ipi vector for PIR
- OS-6848 vm_localize_resources() CPU check is wrong
- OS-6829 bhyve can miss PIR wake-ups
- OS-6814 OS-6726 missed exception_lists/check_rtime
- OS-6820 bhyve provisioning times out after OS-6720
- OS-6720 uart-ignorant guest sends bhyve into a spin
- OS-6715 extraneous rxfifo_available() in uart_sock_drain()
- OS-6694 bhyve uart emulation does not handle 4 byte requests
- OS-6628 bhyve needs to support at least 32 vcpus
- OS-6639 bhyve memory allocations that can't succeed should fail or at least be interruptible
- OS-6716 bhyve guest VCPUs pause intermittently

Related issues

Has duplicate illumos gate - Feature #12734: bhyve on AMD should report SVM as disabledClosed

Actions #1

Updated by Patrick Mooney about 2 years ago

On top of all the testing that was done in SmartOS (and later OmniOSce), I smoke tested this along with the viona bits in order to get an end-to-end experience on hardware. An example Linux guest was able to boot just like it would under SmartOS bhyve.

Actions #2

Updated by Patrick Mooney about 2 years ago

  • Description updated (diff)
Actions #3

Updated by Patrick Mooney about 2 years ago

  • Has duplicate Feature #12734: bhyve on AMD should report SVM as disabled added
Actions #4

Updated by Electric Monk about 2 years ago

  • Status changed from In Progress to Closed
  • % Done changed from 0 to 100

git commit 4c87aefe8930bd07275b8dd2e96ea5f24d93a52e

commit  4c87aefe8930bd07275b8dd2e96ea5f24d93a52e
Author: Patrick Mooney <>
Date:   2020-05-15T18:37:56.000Z

    12665 want modern bhyve
    Portions contributed by: Hans Rosenfeld <>
    Portions contributed by: John Levon <>
    Portions contributed by: Mike Gerdts <>
    Portions contributed by: Andy Fiddaman <>
    Portions contributed by: Dominik Hassler <>
    Portions contributed by: Jerry Jelinek <>
    Portions contributed by: Robert Mustacchi <>
    Portions contributed by: Mike Zeller <>
    Reviewed by: Andy Fiddaman <>
    Approved by: Dan McDonald <>


Also available in: Atom PDF