Bug #12936
closed
bhyve vlapic needs ability to bypass isrvec_stk
100%
Description
The emulated APIC in bhyve (vlapic) contains a data structure isrvec_stk which tracks the vectors of delivered interrupts for the purpose of asserting certain invariants. While the use of ifdef around pieces relating to isrvec_stk might make it seem as if it's just an additional check on top of the standard logic, it is (via vlapic_update_ppr) load bearing. The same data could be effectively derived from the isr0-7 registers in the LAPIC page. Considering that the state contained in isrvec_stk is a bhyve implementation detail, rather than architecturally defined, it would be nice if it could be omitted. This is especially true for save/restore or live-migrate cases, where that isrvec_stk data would need to be rehydrated and/or its checks disabled, lest the new host running the VM fail an assert because the architectural and implementation states differ.
The logic itself could be kept around for debugging purposes, but in a way that makes it optional.
Updated by
Updated by Patrick Mooney about 1 year ago
It seems that few OSes use nested interrupt priorities these days. While illumos uses the TPR (unlike Linux) to elevate thread priority during an interrupt, it does immediately EOI vectors, using the TPR to set priority rather than the active ISR. OpenBSD, as it turns out, does still use the priority set by ISR in addition to allowing interrupt preemption. By testing it with APICv features turned off, it was an opportunity to verify that both the "hot path" EOI vector selection works, as well as the optional isrvec_stk logic. All of this appeared to be working fine while driving both disk and network loads in an OpenBSD 6.7 VM. With dtrace, I could see that isrvec_stk_top in the debug/verification logic rose as high as 3, indicating two preempted ISRs.
Updated by Patrick Mooney about 1 year ago
Windows Server 2019 was another test platform for nested priorities, where I saw a similar 3-deep isrvec_stk_top value when processing an EOI.
Updated by Patrick Mooney about 1 year ago
I booted up Linux, FreeBSD, and OmniOS guests with the same debugging, but all of those seemed to use the APIC in such a way that only one vector would be pending at any given time.
Updated by Patrick Mooney about 1 year ago
A derivative of this change was tested by a member of the FreeBSD community in their bhyve environment with a variety of guests. It appeared to have no issues under load.
Updated by Electric Monk about 1 year ago
- Status changed from New to Closed
- % Done changed from 0 to 100
git commit 5365b8a5984eca60a3cedc7f9f738e51fb056bec
commit 5365b8a5984eca60a3cedc7f9f738e51fb056bec
Author: Patrick Mooney <pmooney@pfmooney.com>
Date: 2020-09-04T14:45:15.000Z
12936 bhyve vlapic needs ability to bypass isrvec_stk
Reviewed by: Mike Zeller <mike.zeller@joyent.com>
Reviewed by: Robert Mustacchi <rm@fingolfin.org>
Approved by: Dan McDonald <danmcd@joyent.com>