Bug #13008


bhyve devmem could match better

Added by Patrick Mooney almost 2 years ago. Updated almost 2 years ago.

Start date:
Due date:
% Done:


Estimated time:
Gerrit CR:


One type of guest memory associated with a bhyve instance is referred to as "devmem". These regions are used primarily for making the bootrom contents available at the expected address (but read-only to the guest). On FreeBSD, they are mmap()-able via a separate device entry from the main VM, but on illumos, that would be a fair bit of effort, so we play some games to make it accessible. A comment in the source describes how:

 * On native FreeBSD, bhyve consumers are allowed to create 'devmem' segments
 * in the vm which appear with their own name related to the vm under /dev.
 * Since this would be a hassle from an sdev perspective and would require a
 * new cdev interface (or complicate the existing one), we choose to implement
 * this in a different manner.  When 'devmem' mappings are created, an
 * identifying off_t is communicated back out to userspace.  That off_t,
 * residing above the normal guest memory space, can be used to mmap the
 * 'devmem' mapping from the already-open vm device.

Basically, when userspace wants to mmap() in a devmem segment, it uses an ioctl with the segment name to learn of the "magic" offset. When the map is occurring, bhyve attempts to find the segment with the size/offset, but is rather strict in its search:

static boolean_t
vmmdev_devmem_segid(vmm_softc_t *sc, off_t off, off_t len, int *segidp)
        list_t *dl = &sc->vmm_devmem_list;
        vmm_devmem_entry_t *de = NULL;

        VERIFY(off >= VM_DEVMEM_START);

        for (de = list_head(dl); de != NULL; de = list_next(dl, de)) {
                /* XXX: Only hit on direct offset/length matches for now */
                if (de->vde_off == off && de->vde_len == len) {

This was working fine, but with the changes in #12792, the segment created for the bootrom is no longer sized relative to the bootrom file. It is, instead, sized to the bootrom maximum of 16MB. If mdb-bhyve is used to attempt to inspect a given instance, it goes to map the guest memory and its devmem segments. In doing so, it uses the mapped size, rather than the segment size, for the 'bootrom' segment, and fails to map:

mdb -b vmname
mdb: failed to map vmname: No such device

Either mdb-bhyve could use the segment size and/or vmmdev_devmem_segid could be made less strict in its matching checks. The latter option seems more correct in the long run.

Actions #1

Updated by Electric Monk almost 2 years ago

  • Gerrit CR set to 827
Actions #2

Updated by Patrick Mooney almost 2 years ago

With the proposed fix, I'm able to attach to bhyve instances with mdb -b without encountering the 'No such device' mapping failure.

Actions #3

Updated by Electric Monk almost 2 years ago

  • Status changed from In Progress to Closed
  • % Done changed from 0 to 100

git commit c3d209cab1511045e9bb1a521f1bd85995d4fd7e

commit  c3d209cab1511045e9bb1a521f1bd85995d4fd7e
Author: Patrick Mooney <>
Date:   2020-08-10T22:55:03.000Z

    13008 bhyve devmem could match better
    13009 mdb-bhyve mishandles memseg offsets
    13010 bhyve should not exit when VM debugged
    Reviewed by: Andy Fiddaman <>
    Reviewed by: Hans Rosenfeld <>
    Approved by: Robert Mustacchi <>


Also available in: Atom PDF