illumos gate

Doing instruction decoding and emulation for MMIO is a necessary task for bhyve. Unlike the in/out instructions, which have decoding acceleration and relatively simple semantics, MMIO can be performed by a vast array of instructions for which there is no easy acceleration in SVM or VMX. This means manually decoding the instruction to determine the MMIO address(es), performing the emulated MMIO (in kernel or userspace) and then applying the result to registers and/or memory. There is an existing system for this in bhyve, implemented in vmm_instruction_emul.c. When MMIO cannot be fulfilled in the kernel, bhyve traps out to userspace where it's expected that a virtual device will be present to service the request. It is that interface boundary which is the problem. Rather that communicating the MMIO request (GPA, size, operation) to userspace, so that its result can be injected back into the kernel emulation logic to be applied, the current logic copies the entire virtual instruction emulation context down to userspace. The vmm_instruction_emul.c code is compiled there as well, so it can request the MMIO and apply the effects in userspace, using the struct vie blob there. This effectively exposes the implementation details of the MMIO emulation to userspace, forcing any consumer there to either handle the struct vie blob, or fetch and decode the instruction from scratch themselves. Furthermore, it means that any changes in structure to vmm_emulation_emul.c need to be compatible both with the kernel and userspace.

I propose that the instruction emulation logic be updated to emit MMIO requests to userspace, rather than copied down the entire struct vie blob. Then userspace can fulfill the MMIO request on VM entry, and the decoded instruction state (kept in the kernel) can be used to finish the emulation. This frees userspace from the complicated obligation of applying the MMIO result itself and more cleanly separates userspace and the bhyve kernel interfaces.