illumos gate

One type of guest memory associated with a bhyve instance is referred to as "devmem". These regions are used primarily for making the bootrom contents available at the expected address (but read-only to the guest). On FreeBSD, they are mmap()-able via a separate device entry from the main VM, but on illumos, that would be a fair bit of effort, so we play some games to make it accessible. A comment in the source describes how:

 * On native FreeBSD, bhyve consumers are allowed to create 'devmem' segments
 * in the vm which appear with their own name related to the vm under /dev.
 * Since this would be a hassle from an sdev perspective and would require a
 * new cdev interface (or complicate the existing one), we choose to implement
 * this in a different manner.  When 'devmem' mappings are created, an
 * identifying off_t is communicated back out to userspace.  That off_t,
 * residing above the normal guest memory space, can be used to mmap the
 * 'devmem' mapping from the already-open vm device.

Basically, when userspace wants to mmap() in a devmem segment, it uses an ioctl with the segment name to learn of the "magic" offset. When the map is occurring, bhyve attempts to find the segment with the size/offset, but is rather strict in its search:

static boolean_t
vmmdev_devmem_segid(vmm_softc_t *sc, off_t off, off_t len, int *segidp)
{
        list_t *dl = &sc->vmm_devmem_list;
        vmm_devmem_entry_t *de = NULL;

        VERIFY(off >= VM_DEVMEM_START);

        for (de = list_head(dl); de != NULL; de = list_next(dl, de)) {
                /* XXX: Only hit on direct offset/length matches for now */
                if (de->vde_off == off && de->vde_len == len) {
                        break;
                }
        }

This was working fine, but with the changes in #12792, the segment created for the bootrom is no longer sized relative to the bootrom file. It is, instead, sized to the bootrom maximum of 16MB. If mdb-bhyve is used to attempt to inspect a given instance, it goes to map the guest memory and its devmem segments. In doing so, it uses the mapped size, rather than the segment size, for the 'bootrom' segment, and fails to map:

mdb -b vmname
mdb: failed to map vmname: No such device

Either mdb-bhyve could use the segment size and/or vmmdev_devmem_segid could be made less strict in its matching checks. The latter option seems more correct in the long run.