Project

General

Profile

Actions

Bug #13016

closed

bhyve cannot emulate MMIO from bootrom

Added by Patrick Mooney over 1 year ago. Updated over 1 year ago.

Status:
Closed
Priority:
Normal
Category:
bhyve
Start date:
Due date:
% Done:

100%

Estimated time:
Difficulty:
Medium
Tags:
bhyve
Gerrit CR:

Description

While testing a new build of the UEFI ROM on an older Intel machine, andyf was observing bhyve exit on EFAULT:

vm_run error -1, errno 14

After some dtracing, this was found to be emitted by the instruction emulation logic while it attempted to fetch a MMIO instruction directly from the bootrom:

1  68812             vm_copy_setup:return                14
1  68896     vmm_fetch_instruction:return                14
1  67872       vm_handle_inst_emul:return                14

Further digging confirmed that %rip pointed to a mov instruction which was attempting to read from the APIC.

0xfffcdd2e:                     movl   (%rbx),%esi
%rbx = 0x00000000fee000f0

This is a little bit strange. Surely the uefi bootrom, old and new, has been accessing MMIO resources during bootup. It does appear as if the ROM relocates its text into "normal" system DRAM before getting into the bulk of its work, so perhaps the new build has changed the order of certain component initialization such that it now accesses these APIC resources before the relocation occurs.

With that concern out of the way, let's turn our eye to why it cannot fetch the instruction for decoding. Further tracing revealed that vm_gpa_hold was failing during the instruction fetch. Looking at its source, the reason becomes apparent:

...

        count = 0;
        for (i = 0; i < VM_MAX_MEMMAPS; i++) {
                mm = &vm->mem_maps[i];
                if (sysmem_mapping(vm, mm) && gpa >= mm->gpa &&
                    gpa < mm->gpa + mm->len) {
                        count = vm_fault_quick_hold_pages(&vm->vmspace->vm_map,
                            trunc_page(gpa), PAGE_SIZE, reqprot, &m, 1);
                        break;
                }
        }
...

The bootrom, being a somewhat special resource, is not considered "sysmem" (unlike the normal physical memory presented to the guest). For that reason, bhyve is not allowing a hold be placed on that GPA before the copy is initiated. Looking at the bhyve memory map logic, there's no clear reason why this is enforced. We should be able to lift this specific limitation.

It should be noted that this failure was found only on Intel hardware without APICv support. With APICv, the accesses performed by the bootrom prior to relocation were able to be accelerated, so no instruction emulation was required. I reproduced this on an APICv-capable machine by disabling APICv via vmx_capabilities.

Actions

Also available in: Atom PDF