illumos gate

There are two places in this function (on i86pc) that we write to the APIC command register to generate IPIs; the first is:

apic_reg_ops->apic_write(APIC_INT_CMD1,
    AV_NMI | AV_LEVEL | AV_SH_ALL_EXCSELF);

which is done unconditionally regardless of the command we've received to cause all other CPUs to shut down their APIC code and stop doing work. Unfortunately this is documented in the AMD64 architecture manual to be invalid; see table 16-4 in volume 2, revision 4.03. When TM = level, Level must be set to asserted as well. In fact, all NMIs are delivered in edge mode and we probably shouldn't be setting AV_LEVEL at all. If we do, however, we should set AV_ASSERT rather than AV_LEVEL. This appears to work regardless, at least on some machines, but it certainly isn't guaranteed given that it directly contradicts the manual.

The second place is in the AD_FASTREBOOT path:

apic_reg_ops->apic_write(APIC_INT_CMD1,
    AV_ASSERT | AV_RESET | AV_SH_ALL_EXCSELF);

Here we're correctly using AV_ASSERT instead of AV_LEVEL to send an INIT message (not reset, which doesn't exist). While some documentation suggests that this message is deprecated on some AMD processors, it is at least architecturally defined and this usage appears correct.

It's not clear whether this bug has any real-world impact, or which processors might misbehave because of it. AMD Milan processors seem to do the right thing regardless of whether we use AV_LEVEL or AV_ASSERT here. But since our clear intent is to assert this IPI, not to deassert it, it seems reasonable to switch to either AV_ASSERT or edge-triggered delivery.