illumos gate

This change is part of a series (including #14635) of work to enable save/restore and eventually live migration in bhyve (via Propolis). By itself, it is inadequate to demonstrate even save/restore with success, given the additional state which needs to be moved. It was split from from #14635 in order to make it easier to review. When built with #14635 floated on top, and run with an experimental patchset of Propolis, we were able to successfully pause a running bhyve instance, dump its state to a file, destroy the in-kernel resources, exit the initial Propolis process, and then restore the guest in a new Propolis process with a fresh in-kernel VMM instance rehydrated with the saved information (DRAM contents and device state)

While this functionality is proven out, the riskiest portion of the added behavior (the ability to load device state into an existing instance) is being kept hidden behind a kernel variable (vmm_allow_state_writes) which must be changed via mdb -kw in order to be used. This allows developers and adventurous testers to use the functionality at their discretion while keeping it masked from other users until we are more comfortable about its robustness.

On a system running bits built from the change, the bhyve test suite still passes (on both AMD and Intel):

Test: /opt/bhyve-tests/tests/mevent/vnode_zvol (run as root)      [00:02] [PASS]
Test: /opt/bhyve-tests/tests/inst_emul/rdmsr (run as root)        [00:00] [PASS]
Test: /opt/bhyve-tests/tests/inst_emul/wrmsr (run as root)        [00:00] [PASS]
Test: /opt/bhyve-tests/tests/inst_emul/triple_fault (run as root) [00:00] [PASS]
Test: /opt/bhyve-tests/tests/kdev/vatpit_freq (run as root)       [00:00] [PASS]
Test: /opt/bhyve-tests/tests/kdev/vhpet_freq (run as root)        [00:00] [PASS]
Test: /opt/bhyve-tests/tests/kdev/vlapic_freq (run as root)       [00:00] [PASS]
Test: /opt/bhyve-tests/tests/kdev/vlapic_freq_periodic (run as root) [00:00] [PASS]
Test: /opt/bhyve-tests/tests/kdev/vlapic_mmio_access (run as root) [00:00] [PASS]
Test: /opt/bhyve-tests/tests/kdev/vlapic_msr_access (run as root) [00:00] [PASS]
Test: /opt/bhyve-tests/tests/kdev/vpmtmr_freq (run as root)       [00:00] [PASS]
Test: /opt/bhyve-tests/tests/mevent/lists_delete (run as root)    [00:00] [PASS]
Test: /opt/bhyve-tests/tests/mevent/read_disable (run as root)    [00:00] [PASS]
Test: /opt/bhyve-tests/tests/mevent/read_pause (run as root)      [00:00] [PASS]
Test: /opt/bhyve-tests/tests/mevent/read_requeue (run as root)    [00:00] [PASS]
Test: /opt/bhyve-tests/tests/mevent/vnode_file (run as root)      [00:09] [PASS]
Test: /opt/bhyve-tests/tests/vmm/fpu_getset (run as root)         [00:00] [PASS]
Test: /opt/bhyve-tests/tests/vmm/interface_version (run as root)  [00:00] [PASS]
Test: /opt/bhyve-tests/tests/vmm/mem_devmem (run as root)         [00:00] [PASS]
Test: /opt/bhyve-tests/tests/vmm/mem_partial (run as root)        [00:00] [PASS]
Test: /opt/bhyve-tests/tests/vmm/mem_seg_map (run as root)        [00:00] [PASS]

Results Summary
PASS      21

Running Time:   00:00:14
Percent passed: 100.0%
Log directory:  /var/tmp/test_results/20220623T031540

The standard battery of test guests all booted and ran (on AMD and Intel) as well.

When testing with #14635 (and dev propolis), if vmm_allow_state_writes was written to non-zero prior to attempting a VM restore, the VMM_DATA_WRITE operation attempts would fail as expected.