illumos gate

Today, when guest actions lead to a SIPI being issued by a vlapic to one of the not-yet-active vCPUs, this predicament is communicated to userspace via the VM_EXITCODE_SPINUP_AP code (the relevant details stashed in the vmexit struct as well). This vmexit state is "injected" directly into the vcpu state in vlapic_icrlo_write_handler:

        if (mode == APIC_DELMODE_STARTUP) {
                if (vlapic->vcpuid == 0 && dest != 0 && dest < maxcpus) {
                        vlapic2 = vm_lapic(vlapic->vm, dest);

                        /*
                         * Ignore SIPIs in any state other than wait-for-SIPI
                         */
                        if (vlapic2->boot_state != BS_SIPI)
                                return (0);

                        vlapic2->boot_state = BS_RUNNING;

                        *retu = true;
                        vmexit = vm_exitinfo(vlapic->vm, vlapic->vcpuid);
                        vmexit->exitcode = VM_EXITCODE_SPINUP_AP;
                        vmexit->u.spinup_ap.vcpu = dest;
                        vmexit->u.spinup_ap.rip = vec << PAGE_SHIFT;

                        return (0);
                }
        }

This strikes me as a pretty blatant layering violation of the API. Furthermore, the KERNEMU calls added recently by upstream bhyve are done outside of VM_RUN context, and would effectively lose this SIPI data if userspace attempted to emulate an ICRLO write which initiated a SIPI. It would probably make sense for the vlapic to stash SIPI state so it could be queried later. The in-kernel emulation could use this to set vmexit fields at an appropriate time after performing vlapic writes for mmio/wrmsr. If a KERNEMU write generated such a SIPI, vm_run() could detect it in the vlapic state and immediately issue the appropriate vmexit.