Bug #12989


improve interface boundary for bhyve MMIO

Added by Patrick Mooney almost 2 years ago. Updated almost 2 years ago.

Start date:
Due date:
% Done:


Estimated time:
Gerrit CR:


Doing instruction decoding and emulation for MMIO is a necessary task for bhyve. Unlike the in/out instructions, which have decoding acceleration and relatively simple semantics, MMIO can be performed by a vast array of instructions for which there is no easy acceleration in SVM or VMX. This means manually decoding the instruction to determine the MMIO address(es), performing the emulated MMIO (in kernel or userspace) and then applying the result to registers and/or memory. There is an existing system for this in bhyve, implemented in vmm_instruction_emul.c. When MMIO cannot be fulfilled in the kernel, bhyve traps out to userspace where it's expected that a virtual device will be present to service the request. It is that interface boundary which is the problem. Rather that communicating the MMIO request (GPA, size, operation) to userspace, so that its result can be injected back into the kernel emulation logic to be applied, the current logic copies the entire virtual instruction emulation context down to userspace. The vmm_instruction_emul.c code is compiled there as well, so it can request the MMIO and apply the effects in userspace, using the struct vie blob there. This effectively exposes the implementation details of the MMIO emulation to userspace, forcing any consumer there to either handle the struct vie blob, or fetch and decode the instruction from scratch themselves. Furthermore, it means that any changes in structure to vmm_emulation_emul.c need to be compatible both with the kernel and userspace.

I propose that the instruction emulation logic be updated to emit MMIO requests to userspace, rather than copied down the entire struct vie blob. Then userspace can fulfill the MMIO request on VM entry, and the decoded instruction state (kept in the kernel) can be used to finish the emulation. This frees userspace from the complicated obligation of applying the MMIO result itself and more cleanly separates userspace and the bhyve kernel interfaces.

Related issues

Related to illumos gate - Bug #13139: bhyve bungles math for VMX ins/outs sizeClosedPatrick Mooney

Actions #1

Updated by Electric Monk almost 2 years ago

  • Gerrit CR set to 815
Actions #2

Updated by Patrick Mooney almost 2 years ago

To test, I booted up the following guests to make sure that they functioned properly:
- CentOS 8
- FreeBSD-CURRENT (a semi-recent build)
- OpenBSD 6.7
- OmniOSCE r34
- Windows Server 2019

In each case, the instance booted up normally to the point I could log in and issue a shutdown.
Those tests were repeated on both Intel and AMD systems.

Actions #3

Updated by Electric Monk almost 2 years ago

  • Status changed from In Progress to Closed
  • % Done changed from 0 to 100

git commit e0c0d44e917080841514d0dd031a696c74e8c435

commit  e0c0d44e917080841514d0dd031a696c74e8c435
Author: Patrick Mooney <>
Date:   2020-08-20T19:19:51.000Z

    12989 improve interface boundary for bhyve MMIO
    12990 improve interface boundary for bhyve ins/outs
    12991 bhyve vlapic should SIPI more carefully
    Reviewed by: Mike Zeller <>
    Reviewed by: Joshua M. Clulow <>
    Reviewed by: Robert Mustacchi <>
    Approved by: Gordon Ross <>

Actions #4

Updated by Patrick Mooney almost 2 years ago

  • Related to Bug #13139: bhyve bungles math for VMX ins/outs size added

Also available in: Atom PDF