Bug #13275


bhyve needs richer INIT/SIPI support

Added by Patrick Mooney over 1 year ago. Updated over 1 year ago.

Start date:
Due date:
% Done:


Estimated time:
Gerrit CR:


Today, bhyve's support for sending INIT and SIPI messages from the APIC is extremely limited. It only supports single-destination messages originating from the BSP. Furthermore, it has no support for re-INIT-ing a vCPU after it has started. In order to run newer versions of the UEFI ROM, or anything else that depends on the vLAPIC meeting its architecturally defined expectations, bhyve should support more complex INIT/SIPI actions. This means both re-INIT-ing (and subsequently SIPI-ing) vCPUs while the system is running, as well as being able to use the all-excluding-self destination shorthand.

Related issues

Related to illumos gate - Bug #13645: bhyve queues wrong SIPIClosedPatrick Mooney

Actions #1

Updated by Electric Monk over 1 year ago

  • Gerrit CR set to 1035
Actions #2

Updated by Patrick Mooney over 1 year ago

The primary test case for this is running guests under an alternative bhyve userspace which makes use of a stock OVMF bootrom. That ROM takes all the vCPUs through INIT/SIPI (notably using the all-but-self destination shorthand) prior to running the bootloader, leaving the guest OS to take them through another lap through those INIT/SIPI steps. With that more complex workflow verified, "traditional" guests using the old bhyve-specific bootrom were also booted to ensure the old behavior (which amounted to a subset of the OVMF behavior) proceeded as expected.

Scrutiny of boot dmesg from a CentOS guest showed the (guest) kernel grumbling about seemingly spurious interrupts during testing. This was traced back to (PIT) external interrupt state which was not being properly cleared by the INIT logic. This lead to the vCPU reset triggered by INIT being much more comprehensive (in line with architecturally defined expectations), including the APIC and ExtINT/exception state. With that added, CentOS no longer complained about the interrupts and booted as cleanly as before.

Actions #3

Updated by Patrick Mooney over 1 year ago

The usual collection of guest OSes were used to smoke-test the INIT/SIPI sequence as those VMs booted and brought APs online. This was performed on both AMD and Intel hardware (the latter uncovering some exitcode handling changes which needed to be integrated into the wad). With the proposed patch in place, they all booted and ran as expected, with their behavior appearing no different than on stock bits.

Actions #4

Updated by Patrick Mooney over 1 year ago

I also re-tested the latest version of this wad with the experimental userspace (and OVMF bootrom) to verify that it still behaves as desired when booting a guest. It functioned properly, as expected.

Actions #5

Updated by Electric Monk over 1 year ago

  • Status changed from In Progress to Closed
  • % Done changed from 0 to 100

git commit 2606939d92dd3044a9851b2930ebf533c3c03892

commit  2606939d92dd3044a9851b2930ebf533c3c03892
Author: Patrick Mooney <>
Date:   2021-01-08T22:12:27.000Z

    13275 bhyve needs richer INIT/SIPI support
    Reviewed by: Robert Mustacchi <>
    Approved by: Gordon Ross <>

Actions #6

Updated by Patrick Mooney about 1 year ago

  • Related to Bug #13645: bhyve queues wrong SIPI added

Also available in: Atom PDF