Bug #14456


bhyve needs fpu import/export

Added by Patrick Mooney 7 months ago. Updated 6 months ago.

Start date:
Due date:
% Done:


Estimated time:
Gerrit CR:


One of the pieces of in-kernel bhyve state which cannot be manipulated during the lifetime of a guest instance is the FPU. While the VMM faithfully restores and saves its contents on guest entry and exit (respectively), there is no way to access that saved state from userspace, or inject a different state to restore on a later run (as would be needed for migration of a guest). We need an interface for userspace to import and export vCPU FPU state. Care must be taken while doing so, as XRSTOR will emit #gp if the to-be-loaded data is invalid. When importing data from userspace, we must validate it with the protection of on_trap/on_fault to ensure that it will not cause problems when being loaded for the guest. It would also be nice if the data structures for import/export had adequate associated metadata to communicate capabilities (SSE, AVX, AVX512) and sizing of the involved data.

Actions #1

Updated by Electric Monk 6 months ago

  • Gerrit CR set to 1998
Actions #2

Updated by Patrick Mooney 6 months ago

The first step in testing this was the newly added fpu_getset unit test. In writing it during the development of the actual patch, I was able to ensure that potential fault conditions were exercised. An early mistake where uninitialized memory was passed to it, for example, triggered the #GP behavior before the necessarily on_trap machinery was configured, making it all the more apparent when those issues had been fixed. The unit tests reads and writes data from both the legacy (x87/mmx/sse) area, as well as the AVX area (if the host CPU is capable), including attempts to load invalid data, which the HMA framework must (and does) detect and throw an appropriate error. I hammered the executing of that test while running both the crypto-tests and os-tests suites. They all passed as expected.

While running a guest, I would query its FPU state using the newly-added bhyvectl --get-fpu. When running a workload which utilizes the FPU, such as openssl speed sha1, I would see the register state changing as the calculations progressed.

Actions #3

Updated by Patrick Mooney 6 months ago

The tests noted above were repeated on both AMD and Intel systems (both supporting xsaveopt as their FPU saving mechanism). In addition to the specific tests, I also ran through the normal suite of guest smoke-tests to ensure that the usual suspects would boot/run with the change in place. That all went without any issue.

Actions #4

Updated by Electric Monk 6 months ago

  • Status changed from In Progress to Closed
  • % Done changed from 0 to 100

git commit 957246c9e6c47389c40079995d73eebcc659fb29

commit  957246c9e6c47389c40079995d73eebcc659fb29
Author: Patrick Mooney <>
Date:   2022-02-18T02:23:05.000Z

    14456 bhyve needs fpu import/export
    Reviewed by: Dan Cross <>
    Reviewed by: C Fraire <>
    Reviewed by: Robert Mustacchi <>
    Approved by: Dan McDonald <>


Also available in: Atom PDF