Project

General

Profile

Actions

Bug #13896

closed

bhyve VM interfaces should be better fit

Added by Patrick Mooney almost 2 years ago. Updated over 1 year ago.

Status:
Closed
Priority:
Normal
Category:
bhyve
Start date:
Due date:
% Done:

100%

Estimated time:
Difficulty:
Hard
Tags:
Gerrit CR:
External Bug:

Description

When bhyve was ported to illumos, a makeshift "VM shim" was written to provide the FreeBSD VM interfaces used by bhyve to build/maintain the nested page tables (and associated resources). Keeping source compatibility with upstream bhyve was a priority at that point, which meant replicating FreeBSD interfaces at the cost of simplicity. That area of bhyve ended up being rather circuitous while at the same time being somewhat lacking in capabilities and performance. The current state is a barrier to adding significant features (such as A/D tracking for migration).

As a solution, we propose a significant overhaul of those VM-related interfaces and logic which are consumed by bhyve. The new interfaces should be tailored to what bhyve and its related components need for efficient and straightforward operation.


Related issues

Related to illumos gate - Feature #13833: want bhyve memory reservoirClosedPatrick Mooney

Actions
Related to illumos gate - Bug #13861: bhyve apicv logic could take more careClosedPatrick Mooney

Actions
Related to illumos gate - Bug #13981: bhyve emulation should set dirty bitsClosedPatrick Mooney

Actions
Related to illumos gate - Bug #14251: operationalize bhyve dirty page trackingClosedPatrick Mooney

Actions
Related to illumos gate - Bug #14266: bhyve mishandles TLB flush on VMXClosedPatrick Mooney

Actions
Related to illumos gate - Bug #14367: bhyve gpt mishandles small mappingsClosedPatrick Mooney

Actions
Actions #1

Updated by Patrick Mooney almost 2 years ago

  • Related to Feature #13833: want bhyve memory reservoir added
  • Related to Bug #13861: bhyve apicv logic could take more care added
Actions #2

Updated by Electric Monk almost 2 years ago

  • Gerrit CR set to 1563
Actions #3

Updated by Patrick Mooney almost 2 years ago

  • Related to Bug #13981: bhyve emulation should set dirty bits added
Actions #4

Updated by Patrick Mooney over 1 year ago

A/D tracking (and thus the better interfaces) have been a focus for work done in Propolis, so that has been one of the primary test cases for this code, where platform bits featuring this change, in addition to others, were used as the primary Propolis development and test environment. That also exposed it to more varied usage in terms of VM ioctls and the segment driver, as compared to "vanilla c-bhyve". The bulk of that testing was done with the gpt_track_dirty variable set to its default of false. Some spot-checks of that functionality have been done, and it was observed working, but more heavy testing will be done as more interfaces for migration are built into Propolis. It will remain default-off for the time being, as such tracking of dirty pages does come at a performance cost, and it would be silly to do it when not required.

In addition to the Propolis tests, I ran these bits on both Intel and AMD machines, spinning up the typical battery of bhyve guests for smoke-testing. Windows, Linux, illumos, and OpenBSD guests booted, rebooted, and ran as expected.

Actions #5

Updated by Patrick Mooney over 1 year ago

The ticket for #13981 includes a test program used to exercise the edge case of unmapping an object from the guest-physical address space of a VM which itself is mapped into a userspace process of the host. This is expected to incur a SIGSEGV in that host process, should it attempt to access the now-unmapped guest memory, rather than cause a problem (read: panic) in seg_vmm. Running it under the test bits, it does segfault safely, as expected.

Actions #6

Updated by Patrick Mooney over 1 year ago

I created #14251 to address the need for creating a real interface (beyond mdb -kw) for enabling dirty page tracking on bhyve instances.

Actions #7

Updated by Patrick Mooney over 1 year ago

  • Related to Bug #14251: operationalize bhyve dirty page tracking added
Actions #8

Updated by Electric Monk over 1 year ago

  • Status changed from In Progress to Closed
  • % Done changed from 0 to 100

git commit 0153d828c132fdb1a17c11b99386a3d1b87994cf

commit  0153d828c132fdb1a17c11b99386a3d1b87994cf
Author: Patrick Mooney <pmooney@pfmooney.com>
Date:   2021-11-19T23:00:59.000Z

    13896 bhyve VM interfaces should be better fit
    13981 bhyve emulation should set dirty bits
    Reviewed by: Dan Cross <cross@oxidecomputer.com>
    Reviewed by: Joshua M. Clulow <josh@sysmgr.org>
    Approved by: Dan McDonald <danmcd@joyent.com>

Actions #9

Updated by Patrick Mooney over 1 year ago

  • Related to Bug #14266: bhyve mishandles TLB flush on VMX added
Actions #10

Updated by Patrick Mooney over 1 year ago

  • Related to Bug #14367: bhyve gpt mishandles small mappings added
Actions

Also available in: Atom PDF