illumos gate

While testing new versions of the uefi-edk2 bootrom under bhyve, folks have noted that it runs very slowly during boot. After transferring control to the OS, the guest appears to run without any performance impact. When testing the same ROM on an Intel system, there is no such slowness. Tracing VM exits did not yield any interesting differences between the two systems. Tracing with the profile- probe to sample the guest %rip showed it spending quite a bit of time (15 seconds on an Epyc machine) in the LZMA decompression step of boot-up. After much digging, a culprit was found: The bootrom was setting %cr0.CD, disabling the cache on the CPU. This is not a problem on Intel because the CD (and NW) bits are ignored in the virtual %cr0. It turns out our emulation for AMD is too accurate, and guests expect CD to be a no-op. A further survey of other hypervisors (KVM, VirtualBox, Xen) shows this to be the case.

As a fix, we should perform %cr0 shadowing, similar to how VMX does it. SVM offers some functionality to avoid exits when toggling often-accessed bits like TS. By masking CD (and NW) from the "real" value used by the CPU in guest contest, those newer roms and any other code which assumes cache-disable to be a no-op should function as expected, rather than performing like a CPU from the 90s.