Bug #13016


bhyve cannot emulate MMIO from bootrom

Added by Patrick Mooney almost 2 years ago. Updated almost 2 years ago.

Start date:
Due date:
% Done:


Estimated time:
Gerrit CR:


While testing a new build of the UEFI ROM on an older Intel machine, andyf was observing bhyve exit on EFAULT:

vm_run error -1, errno 14

After some dtracing, this was found to be emitted by the instruction emulation logic while it attempted to fetch a MMIO instruction directly from the bootrom:

1  68812             vm_copy_setup:return                14
1  68896     vmm_fetch_instruction:return                14
1  67872       vm_handle_inst_emul:return                14

Further digging confirmed that %rip pointed to a mov instruction which was attempting to read from the APIC.

0xfffcdd2e:                     movl   (%rbx),%esi
%rbx = 0x00000000fee000f0

This is a little bit strange. Surely the uefi bootrom, old and new, has been accessing MMIO resources during bootup. It does appear as if the ROM relocates its text into "normal" system DRAM before getting into the bulk of its work, so perhaps the new build has changed the order of certain component initialization such that it now accesses these APIC resources before the relocation occurs.

With that concern out of the way, let's turn our eye to why it cannot fetch the instruction for decoding. Further tracing revealed that vm_gpa_hold was failing during the instruction fetch. Looking at its source, the reason becomes apparent:


        count = 0;
        for (i = 0; i < VM_MAX_MEMMAPS; i++) {
                mm = &vm->mem_maps[i];
                if (sysmem_mapping(vm, mm) && gpa >= mm->gpa &&
                    gpa < mm->gpa + mm->len) {
                        count = vm_fault_quick_hold_pages(&vm->vmspace->vm_map,
                            trunc_page(gpa), PAGE_SIZE, reqprot, &m, 1);

The bootrom, being a somewhat special resource, is not considered "sysmem" (unlike the normal physical memory presented to the guest). For that reason, bhyve is not allowing a hold be placed on that GPA before the copy is initiated. Looking at the bhyve memory map logic, there's no clear reason why this is enforced. We should be able to lift this specific limitation.

It should be noted that this failure was found only on Intel hardware without APICv support. With APICv, the accesses performed by the bootrom prior to relocation were able to be accelerated, so no instruction emulation was required. I reproduced this on an APICv-capable machine by disabling APICv via vmx_capabilities.

Actions #1

Updated by Patrick Mooney almost 2 years ago

  • Description updated (diff)
Actions #2

Updated by Electric Monk almost 2 years ago

  • Gerrit CR set to 829
Actions #3

Updated by Patrick Mooney almost 2 years ago

Notes from andyf:

The affected user has also confirmed that this fix works on the old Sandy Bridge machine that was previously unable to boot from the newer ROM.

Actions #4

Updated by Patrick Mooney almost 2 years ago

  • Description updated (diff)
Actions #5

Updated by Electric Monk almost 2 years ago

  • Status changed from In Progress to Closed
  • % Done changed from 0 to 100

git commit bd05d33960c9701232eaa8bdb6a5ce585912ffc6

commit  bd05d33960c9701232eaa8bdb6a5ce585912ffc6
Author: Patrick Mooney <>
Date:   2020-08-05T18:10:04.000Z

    13016 bhyve cannot emulate MMIO from bootrom
    Reviewed by: Andy Fiddaman <>
    Approved by: Dan McDonald <>


Also available in: Atom PDF