Bug #13847


bhyve should handle all APICv exits

Added by Jason King over 2 years ago. Updated over 1 year ago.

Start date:
Due date:
% Done:


Estimated time:
Gerrit CR:
External Bug:


Testing a BE for #13795, booting into a BE caused bhyve to exit with the following:

[2021-06-02T14:20:15.602943000Z]  INFO: zoneadmd/6789 on bob: (stream=stderr)
    vm exit[0]
        reason          VMX
        rip             0xfffffffffbaf03b6
        inst_length     2
        status          0
        exit_reason     44 (APIC access)

[2021-06-02T14:20:15.603164000Z]  INFO: zoneadmd/6789 on bob: (stream=stderr)
        qualification   0x0000000000000204
        inst_type               0
        inst_error              0

This happened after the 'DEBUG enabled' message on the console (so after the kernel was loaded):

illumos Version zfs-crypto-dnode-0-g72d3a7deb1 64-bit
DEBUG enabled

[NOTICE: Zone halted]

Restarting the instance with the same BE, and it booted just fine after that, so I haven't been able to recreate it again, but wanted to capture it in case others hit it.

This was on joyent_20201118T153314Z, so it's not the absolutely most recent, but not terribly ancient either.

I can't be sure this was the same code as when it occurred, but after rebooting, the instruction in question does at least look plausible (specific instruction is noted with asterisks):

> 0xfffffffffbaf03b6::dis
local_apic_read:                pushq  %rbp
local_apic_read+1:              movq   %rsp,%rbp
local_apic_read+4:              pushq  %rdi
local_apic_read+5:              movl   %edi,%edi
local_apic_read+7:              subq   $0x8,%rsp
local_apic_read+0xb:            movq   -0x3bae5c52(%rip),%rax   <apicadr>
local_apic_read+0x12:           leaq   (%rax,%rdi,4),%rax
** local_apic_read+0x16:           movl   (%rax),%eax **
local_apic_read+0x18:           leave
local_apic_read+0x19:           ret
Actions #1

Updated by Jason King over 2 years ago

  • Description updated (diff)
Actions #2

Updated by Patrick Mooney over 2 years ago

A qualification of 204 indicates a linear data read access at offset 204, which would not be a legal access:

Any FP/MMX/SSE access to an APIC register, or any access that touches bytes 4 through 15 of an APIC register may cause undefined behavior and must not be executed.

It's not clear why the guest would have attempted such an access. Without more detail about the system state under the exit, we cannot know for sure. We should probably make the APIC access handlers more robust to emit an expected error to the guest for such behavior, rather than taking an unhandleable exit.

Actions #3

Updated by Patrick Mooney over 1 year ago

  • Assignee set to Patrick Mooney
Actions #4

Updated by Patrick Mooney over 1 year ago

  • Subject changed from bhyve exits on APIC access to bhyve should handle all APICv exits
Actions #5

Updated by Patrick Mooney over 1 year ago

We should probably just fall back to the regular instruction emulation for cases like this.


Also available in: Atom PDF