Project

General

Profile

Actions

Bug #14511

closed

bhyve needs devmem access for all segments

Added by Patrick Mooney 6 months ago. Updated 6 months ago.

Status:
Closed
Priority:
Normal
Category:
bhyve
Start date:
Due date:
% Done:

100%

Estimated time:
Difficulty:
Medium
Tags:
Gerrit CR:

Description

On FreeBSD, bhyve instances enjoy multiple minor nodes from which to provide access to resources associated with a given VM. Chief among them are the memory "segments" (read: objects), which are the blocks of memory which back the guest-physical address space of the instance. While the guest-physical AS itself can be mapped by userspace (and is, indeed, for driver emulation), there are cases where direct access to the underlying memory segments is required. The common case is writing the contents of the "bootrom" segment, which is mapped read-only into the guest-physical AS. The illumos bhyve interface exposes these "devmem" mappings (those which access the memory segment directly, rather than through the guest-physical AS) by placing them at a magic offset in the singular minor node allocated to the instance. Userspace can then query the offset for a given segment id through an ioctl, allowing it to then mmap() it. Presently this is limited to non-"sysmem" segments only -- basically just the bootrom. There is other functionality (such as live migration) which will benefit from those same direct mappings to the "sysmem" objects making up the bulk of guest RAM. We should loosen the restriction so all segments within an instance can be mapped that way.

Actions #1

Updated by Electric Monk 6 months ago

  • Gerrit CR set to 2031
Actions #2

Updated by Patrick Mooney 6 months ago

With the patch applied, the newly written unit tests now passes. Normal bhyve guests can still boot and run as expected. Walking the devmem list associated with the instance shows the lowmem and highmem segments for that guest, in addition to the bootrom:

{
    vde_node = {
        list_next = 0xfffffe5dcb04b400
        list_prev = 0xfffffe59f02abd30
    }
    vde_segid = 0x1
    vde_name = [ '\0', '\0', '\0', '\0', '\0', '\0', '\0', '\0', '\0', '\0', '\0', '\0', '\0', '\0', '\0', '\0', '\0', '\0', '\0', '\0', '\0', '\0', '\0', '\0', '\0', '\0', '\0', '\0', '\0', '\0', '\0', '\0', ... ]
    vde_len = 0x1400000
    vde_off = 0x400000000000
}
{
    vde_node = {
        list_next = 0xfffffe5dcb04b500
        list_prev = 0xfffffe5dcb04b300
    }
    vde_segid = 0
    vde_name = [ '\0', '\0', '\0', '\0', '\0', '\0', '\0', '\0', '\0', '\0', '\0', '\0', '\0', '\0', '\0', '\0', '\0', '\0', '\0', '\0', '\0', '\0', '\0', '\0', '\0', '\0', '\0', '\0', '\0', '\0', '\0', '\0', ... ]
    vde_len = 0xc0000000
    vde_off = 0x400001400000
}
{
    vde_node = {
        list_next = 0xfffffe59f02abd30
        list_prev = 0xfffffe5dcb04b400
    }
    vde_segid = 0x2
    vde_name = [ "bootrom" ]
    vde_len = 0x1000000
    vde_off = 0x4000c1400000
}

The propolis runtime is able to use the new devmem access to sysmem segments in addition to the bootrom.

A dump of the machine after the tests was taken to confirm that no kmem leaks or corruption occurred and no problems on that front were observed.

Actions #3

Updated by Electric Monk 6 months ago

  • Status changed from In Progress to Closed
  • % Done changed from 0 to 100

git commit 3d0662810ae7f231943be2eb0bf9cd8b25496ddb

commit  3d0662810ae7f231943be2eb0bf9cd8b25496ddb
Author: Patrick Mooney <pmooney@pfmooney.com>
Date:   2022-03-02T16:31:35.000Z

    14511 bhyve needs devmem access for all segments
    Reviewed by: Dan Cross <cross@oxidecomputer.com>
    Reviewed by: Luqman Aden <luqman@oxide.computer>
    Reviewed by: Andy Fiddaman <andy@omnios.org>
    Approved by: Robert Mustacchi <rm@fingolfin.org>

Actions

Also available in: Atom PDF