Project

General

Profile

Actions

Bug #13275

closed

bhyve needs richer INIT/SIPI support

Added by Patrick Mooney 6 months ago. Updated 4 months ago.

Status:
Closed
Priority:
Normal
Category:
bhyve
Start date:
Due date:
% Done:

100%

Estimated time:
Difficulty:
Medium
Tags:
Gerrit CR:

Description

Today, bhyve's support for sending INIT and SIPI messages from the APIC is extremely limited. It only supports single-destination messages originating from the BSP. Furthermore, it has no support for re-INIT-ing a vCPU after it has started. In order to run newer versions of the UEFI ROM, or anything else that depends on the vLAPIC meeting its architecturally defined expectations, bhyve should support more complex INIT/SIPI actions. This means both re-INIT-ing (and subsequently SIPI-ing) vCPUs while the system is running, as well as being able to use the all-excluding-self destination shorthand.


Related issues

Related to illumos gate - Bug #13645: bhyve queues wrong SIPIClosedPatrick Mooney

Actions
Actions #1

Updated by Electric Monk 6 months ago

  • Gerrit CR set to 1035
Actions #2

Updated by Patrick Mooney 5 months ago

The primary test case for this is running guests under an alternative bhyve userspace which makes use of a stock OVMF bootrom. That ROM takes all the vCPUs through INIT/SIPI (notably using the all-but-self destination shorthand) prior to running the bootloader, leaving the guest OS to take them through another lap through those INIT/SIPI steps. With that more complex workflow verified, "traditional" guests using the old bhyve-specific bootrom were also booted to ensure the old behavior (which amounted to a subset of the OVMF behavior) proceeded as expected.

Scrutiny of boot dmesg from a CentOS guest showed the (guest) kernel grumbling about seemingly spurious interrupts during testing. This was traced back to (PIT) external interrupt state which was not being properly cleared by the INIT logic. This lead to the vCPU reset triggered by INIT being much more comprehensive (in line with architecturally defined expectations), including the APIC and ExtINT/exception state. With that added, CentOS no longer complained about the interrupts and booted as cleanly as before.

Actions #3

Updated by Patrick Mooney 4 months ago

The usual collection of guest OSes were used to smoke-test the INIT/SIPI sequence as those VMs booted and brought APs online. This was performed on both AMD and Intel hardware (the latter uncovering some exitcode handling changes which needed to be integrated into the wad). With the proposed patch in place, they all booted and ran as expected, with their behavior appearing no different than on stock bits.

Actions #4

Updated by Patrick Mooney 4 months ago

I also re-tested the latest version of this wad with the experimental userspace (and OVMF bootrom) to verify that it still behaves as desired when booting a guest. It functioned properly, as expected.

Actions #5

Updated by Electric Monk 4 months ago

  • Status changed from In Progress to Closed
  • % Done changed from 0 to 100

git commit 2606939d92dd3044a9851b2930ebf533c3c03892

commit  2606939d92dd3044a9851b2930ebf533c3c03892
Author: Patrick Mooney <pmooney@pfmooney.com>
Date:   2021-01-08T22:12:27.000Z

    13275 bhyve needs richer INIT/SIPI support
    Reviewed by: Robert Mustacchi <rm@fingolfin.org>
    Approved by: Gordon Ross <gordon.w.ross@gmail.com>

Actions #6

Updated by Patrick Mooney about 2 months ago

  • Related to Bug #13645: bhyve queues wrong SIPI added
Actions

Also available in: Atom PDF