illumos gate

While debugging an unrelated issue, I was tracing hma_svm_asid_update which bhyve calls to determine if cached TLB entries for a vCPU's ASID need to be flushed on entry to VM context. When the VM is running in steady state, these flushes are not expected to happen often, as the vCPU is not likely be migrating much, and the RVI address space should be unchanging. My tracing showed it to be the opposite: hma_svm_asid_update was returning 0x3 most often, which corresponds to VMCB_FLUSH_ASID. A trace of that function and the npt_flush parameter shines a light on the issue:

hma_svm_asid_update:entry { self->npt_flush = arg2 }
hma_svm_asid_update:return { @[self->npt_flush, arg1] = count() }

... Results from a running machine:
                0                3               25
                0                0             2486
                1                3           143323

It turns out npt_flush appears to be set most of the time. Examination of the call site in bhyve reveals the true problem:

        eptgen = pmap->pm_eptgen;
        flush = hma_svm_asid_update(&vcpustate->hma_asid, flush_by_asid(),
            vcpustate->eptgen == eptgen);

The polarity of the eptgen check is wrong. It should be requesting a flush when the eptgen has changed, not when it is the same. Fortunately the fix, like the bug, is simple: Use !=.