improve interface boundary for bhyve MMIO
Doing instruction decoding and emulation for MMIO is a necessary task for bhyve. Unlike the in/out instructions, which have decoding acceleration and relatively simple semantics, MMIO can be performed by a vast array of instructions for which there is no easy acceleration in SVM or VMX. This means manually decoding the instruction to determine the MMIO address(es), performing the emulated MMIO (in kernel or userspace) and then applying the result to registers and/or memory. There is an existing system for this in bhyve, implemented in
vmm_instruction_emul.c. When MMIO cannot be fulfilled in the kernel, bhyve traps out to userspace where it's expected that a virtual device will be present to service the request. It is that interface boundary which is the problem. Rather that communicating the MMIO request (GPA, size, operation) to userspace, so that its result can be injected back into the kernel emulation logic to be applied, the current logic copies the entire virtual instruction emulation context down to userspace. The
vmm_instruction_emul.c code is compiled there as well, so it can request the MMIO and apply the effects in userspace, using the
struct vie blob there. This effectively exposes the implementation details of the MMIO emulation to userspace, forcing any consumer there to either handle the
struct vie blob, or fetch and decode the instruction from scratch themselves. Furthermore, it means that any changes in structure to
vmm_emulation_emul.c need to be compatible both with the kernel and userspace.
I propose that the instruction emulation logic be updated to emit MMIO requests to userspace, rather than copied down the entire
struct vie blob. Then userspace can fulfill the MMIO request on VM entry, and the decoded instruction state (kept in the kernel) can be used to finish the emulation. This frees userspace from the complicated obligation of applying the MMIO result itself and more cleanly separates userspace and the bhyve kernel interfaces.
Updated by Patrick Mooney almost 2 years ago
To test, I booted up the following guests to make sure that they functioned properly:
- CentOS 8
- FreeBSD-CURRENT (a semi-recent build)
- OpenBSD 6.7
- OmniOSCE r34
- Windows Server 2019
In each case, the instance booted up normally to the point I could log in and issue a shutdown.
Those tests were repeated on both Intel and AMD systems.
Updated by Electric Monk almost 2 years ago
- Status changed from In Progress to Closed
- % Done changed from 0 to 100
commit e0c0d44e917080841514d0dd031a696c74e8c435 Author: Patrick Mooney <firstname.lastname@example.org> Date: 2020-08-20T19:19:51.000Z 12989 improve interface boundary for bhyve MMIO 12990 improve interface boundary for bhyve ins/outs 12991 bhyve vlapic should SIPI more carefully Reviewed by: Mike Zeller <email@example.com> Reviewed by: Joshua M. Clulow <firstname.lastname@example.org> Reviewed by: Robert Mustacchi <email@example.com> Approved by: Gordon Ross <firstname.lastname@example.org>