Project

General

Profile

Actions

Bug #13458

closed

bhyve flushes ASID needlessly

Added by Patrick Mooney 4 months ago. Updated 3 months ago.

Status:
Closed
Priority:
Normal
Category:
bhyve
Start date:
Due date:
% Done:

100%

Estimated time:
Difficulty:
Medium
Tags:
Gerrit CR:

Description

While debugging an unrelated issue, I was tracing hma_svm_asid_update which bhyve calls to determine if cached TLB entries for a vCPU's ASID need to be flushed on entry to VM context. When the VM is running in steady state, these flushes are not expected to happen often, as the vCPU is not likely be migrating much, and the RVI address space should be unchanging. My tracing showed it to be the opposite: hma_svm_asid_update was returning 0x3 most often, which corresponds to VMCB_FLUSH_ASID. A trace of that function and the npt_flush parameter shines a light on the issue:

hma_svm_asid_update:entry { self->npt_flush = arg2 }
hma_svm_asid_update:return { @[self->npt_flush, arg1] = count() }

... Results from a running machine:
                0                3               25
                0                0             2486
                1                3           143323

It turns out npt_flush appears to be set most of the time. Examination of the call site in bhyve reveals the true problem:

        eptgen = pmap->pm_eptgen;
        flush = hma_svm_asid_update(&vcpustate->hma_asid, flush_by_asid(),
            vcpustate->eptgen == eptgen);

The polarity of the eptgen check is wrong. It should be requesting a flush when the eptgen has changed, not when it is the same. Fortunately the fix, like the bug, is simple: Use !=.

Actions #1

Updated by Patrick Mooney 4 months ago

With the comparison corrected, the ASID flush results come out as expected:

                1                3               27
                0                3              105
                0                0           151447

Actions #2

Updated by Electric Monk 4 months ago

  • Gerrit CR set to 1177
Actions #3

Updated by Electric Monk 3 months ago

  • Status changed from In Progress to Closed
  • % Done changed from 0 to 100

git commit 4955144cafa51275edcbd524ec7401038df30387

commit  4955144cafa51275edcbd524ec7401038df30387
Author: Patrick Mooney <pmooney@pfmooney.com>
Date:   2021-01-23T22:06:45.000Z

    13458 bhyve flushes ASID needlessly
    Reviewed by: Andy Fiddaman <andy@omnios.org>
    Reviewed by: Mike Zeller <mike.zeller@joyent.com>
    Approved by: Robert Mustacchi <rm@fingolfin.org>

Actions

Also available in: Atom PDF