Project

General

Profile

Actions

Bug #14732

closed

split vtd out of vmm kernel module

Added by Patrick Mooney 2 months ago. Updated 2 months ago.

Status:
Closed
Priority:
Normal
Category:
bhyve
Start date:
Due date:
% Done:

100%

Estimated time:
Difficulty:
Medium
Tags:
Gerrit CR:

Description

VT-d's use of ACPI necessitates it as a module dependency for bhyve. Considering the IOMMU functionality is a fairly clean boundary line, it would be nice for VT-d (and AMD-VI) to be separately loaded modules. Bhyve could then load them at runtime (if requested), dragging in the additional dependencies.


Related issues

Related to illumos gate - Bug #14742: Support AMD-VI in bhyveNew

Actions
Actions #1

Updated by Electric Monk 2 months ago

  • Gerrit CR set to 2179
Actions #2

Updated by Patrick Mooney 2 months ago

I tested this on and AMD system where AMD-vi is unimplemented, so IOMMU-related requests should fail. Using the test program included in the change, which attempts to query the available IOMMU resource(s), I saw the expected failure:

root@carbide:~# /opt/bhyve-tests/tests/vmm/check_iommu
VMM_CHECK_IOMMU ioctl failed: No such device or address

All of the normal bhyve tests still pass:

Test: /opt/bhyve-tests/tests/mevent/vnode_zvol (run as root)      [00:02] [PASS]
Test: /opt/bhyve-tests/tests/inst_emul/rdmsr (run as root)        [00:00] [PASS]
Test: /opt/bhyve-tests/tests/inst_emul/wrmsr (run as root)        [00:00] [PASS]
Test: /opt/bhyve-tests/tests/inst_emul/triple_fault (run as root) [00:00] [PASS]
Test: /opt/bhyve-tests/tests/kdev/vatpit_freq (run as root)       [00:00] [PASS]
Test: /opt/bhyve-tests/tests/kdev/vhpet_freq (run as root)        [00:00] [PASS]
Test: /opt/bhyve-tests/tests/kdev/vlapic_freq (run as root)       [00:00] [PASS]
Test: /opt/bhyve-tests/tests/kdev/vlapic_freq_periodic (run as root) [00:00] [PASS]
Test: /opt/bhyve-tests/tests/kdev/vlapic_mmio_access (run as root) [00:00] [PASS]
Test: /opt/bhyve-tests/tests/kdev/vlapic_msr_access (run as root) [00:00] [PASS]
Test: /opt/bhyve-tests/tests/kdev/vpmtmr_freq (run as root)       [00:00] [PASS]
Test: /opt/bhyve-tests/tests/mevent/lists_delete (run as root)    [00:00] [PASS]
Test: /opt/bhyve-tests/tests/mevent/read_disable (run as root)    [00:00] [PASS]
Test: /opt/bhyve-tests/tests/mevent/read_pause (run as root)      [00:00] [PASS]
Test: /opt/bhyve-tests/tests/mevent/read_requeue (run as root)    [00:00] [PASS]
Test: /opt/bhyve-tests/tests/mevent/vnode_file (run as root)      [00:09] [PASS]
Test: /opt/bhyve-tests/tests/vmm/fpu_getset (run as root)         [00:00] [PASS]
Test: /opt/bhyve-tests/tests/vmm/interface_version (run as root)  [00:00] [PASS]
Test: /opt/bhyve-tests/tests/vmm/mem_devmem (run as root)         [00:00] [PASS]
Test: /opt/bhyve-tests/tests/vmm/mem_partial (run as root)        [00:00] [PASS]
Test: /opt/bhyve-tests/tests/vmm/mem_seg_map (run as root)        [00:00] [PASS]

Results Summary
PASS      21

Running Time:   00:00:13
Percent passed: 100.0%
Log directory:  /var/tmp/test_results/20220614T211248

Note that check_iommu is not included in the default runfile, since that would cause failures on AMD systems, or intel systems lacking VT-d support.

To test on the Intel side, sjorge installed bits based on this change (built atop OmniOSCE Bloody) on his systems where PCI-passthru is used. He reported that they functioned as expected, with no apparent problems. In fact, an intermittent problem he'd seen where PPT devices occationally didn't attach properly on first try seemed to be absent while he was testing with the updated bits.

Actions #3

Updated by Patrick Mooney 2 months ago

Testing this on my Intel machine (which supports VT-d), the check_iommu test passes:

check_iommu     PASS

As does the rest of the normal suite:

Test: /opt/bhyve-tests/tests/mevent/vnode_zvol (run as root)      [00:02] [PASS]
Test: /opt/bhyve-tests/tests/inst_emul/rdmsr (run as root)        [00:00] [PASS]
Test: /opt/bhyve-tests/tests/inst_emul/wrmsr (run as root)        [00:00] [PASS]
Test: /opt/bhyve-tests/tests/inst_emul/triple_fault (run as root) [00:00] [PASS]
Test: /opt/bhyve-tests/tests/kdev/vatpit_freq (run as root)       [00:00] [PASS]
Test: /opt/bhyve-tests/tests/kdev/vhpet_freq (run as root)        [00:00] [PASS]
Test: /opt/bhyve-tests/tests/kdev/vlapic_freq (run as root)       [00:00] [PASS]
Test: /opt/bhyve-tests/tests/kdev/vlapic_freq_periodic (run as root) [00:00] [PASS]
Test: /opt/bhyve-tests/tests/kdev/vlapic_mmio_access (run as root) [00:00] [PASS]
Test: /opt/bhyve-tests/tests/kdev/vlapic_msr_access (run as root) [00:00] [PASS]
Test: /opt/bhyve-tests/tests/kdev/vpmtmr_freq (run as root)       [00:00] [PASS]
Test: /opt/bhyve-tests/tests/mevent/lists_delete (run as root)    [00:00] [PASS]
Test: /opt/bhyve-tests/tests/mevent/read_disable (run as root)    [00:00] [PASS]
Test: /opt/bhyve-tests/tests/mevent/read_pause (run as root)      [00:00] [PASS]
Test: /opt/bhyve-tests/tests/mevent/read_requeue (run as root)    [00:00] [PASS]
Test: /opt/bhyve-tests/tests/mevent/vnode_file (run as root)      [00:09] [PASS]
Test: /opt/bhyve-tests/tests/vmm/fpu_getset (run as root)         [00:00] [PASS]
Test: /opt/bhyve-tests/tests/vmm/interface_version (run as root)  [00:00] [PASS]
Test: /opt/bhyve-tests/tests/vmm/mem_devmem (run as root)         [00:00] [PASS]
Test: /opt/bhyve-tests/tests/vmm/mem_partial (run as root)        [00:00] [PASS]
Test: /opt/bhyve-tests/tests/vmm/mem_seg_map (run as root)        [00:00] [PASS]

Results Summary
PASS      21

Running Time:   00:00:15
Percent passed: 100.0%
Log directory:  /var/tmp/test_results/20220614T223354

Since I lack a setup for doing testing PPT functionality, I didn't exercise those bits (since sjorge already had).

Actions #4

Updated by Patrick Mooney 2 months ago

  • Related to Bug #14742: Support AMD-VI in bhyve added
Actions #5

Updated by Electric Monk 2 months ago

  • Status changed from In Progress to Closed
  • % Done changed from 0 to 100

git commit e760f15095bdc9fa107e7c20ed2a5e4fb5865c1d

commit  e760f15095bdc9fa107e7c20ed2a5e4fb5865c1d
Author: Patrick Mooney <pmooney@pfmooney.com>
Date:   2022-06-15T00:21:54.000Z

    14732 split vtd out of vmm kernel module
    Reviewed by: Dan Cross <cross@oxidecomputer.com>
    Reviewed by: Jorge Schrauwen <sjorge@blackdot.be>
    Reviewed by: Andy Fiddaman <andy@omnios.org>
    Approved by: Dan McDonald <danmcd@mnx.io>

Actions

Also available in: Atom PDF