bhyve needs richer INIT/SIPI support
Today, bhyve's support for sending INIT and SIPI messages from the APIC is extremely limited. It only supports single-destination messages originating from the BSP. Furthermore, it has no support for re-INIT-ing a vCPU after it has started. In order to run newer versions of the UEFI ROM, or anything else that depends on the vLAPIC meeting its architecturally defined expectations, bhyve should support more complex INIT/SIPI actions. This means both re-INIT-ing (and subsequently SIPI-ing) vCPUs while the system is running, as well as being able to use the all-excluding-self destination shorthand.
Updated by Patrick Mooney 5 months ago
The primary test case for this is running guests under an alternative bhyve userspace which makes use of a stock OVMF bootrom. That ROM takes all the vCPUs through INIT/SIPI (notably using the all-but-self destination shorthand) prior to running the bootloader, leaving the guest OS to take them through another lap through those INIT/SIPI steps. With that more complex workflow verified, "traditional" guests using the old bhyve-specific bootrom were also booted to ensure the old behavior (which amounted to a subset of the OVMF behavior) proceeded as expected.
Scrutiny of boot dmesg from a CentOS guest showed the (guest) kernel grumbling about seemingly spurious interrupts during testing. This was traced back to (PIT) external interrupt state which was not being properly cleared by the INIT logic. This lead to the vCPU reset triggered by INIT being much more comprehensive (in line with architecturally defined expectations), including the APIC and ExtINT/exception state. With that added, CentOS no longer complained about the interrupts and booted as cleanly as before.
Updated by Patrick Mooney 4 months ago
The usual collection of guest OSes were used to smoke-test the INIT/SIPI sequence as those VMs booted and brought APs online. This was performed on both AMD and Intel hardware (the latter uncovering some exitcode handling changes which needed to be integrated into the wad). With the proposed patch in place, they all booted and ran as expected, with their behavior appearing no different than on stock bits.
Updated by Electric Monk 4 months ago
- Status changed from In Progress to Closed
- % Done changed from 0 to 100
commit 2606939d92dd3044a9851b2930ebf533c3c03892 Author: Patrick Mooney <email@example.com> Date: 2021-01-08T22:12:27.000Z 13275 bhyve needs richer INIT/SIPI support Reviewed by: Robert Mustacchi <firstname.lastname@example.org> Approved by: Gordon Ross <email@example.com>