illumos gate

Historically, pcitool has been used as a way to try and get information about PCI devices, but is undocumented, most pieces of it intended to be private, and hard to use. To that end, this adds a new, temporarily private command, pcieadm. This has three current sub-commands implemented:

• show-devs which allows you to see a summary of PCI devices and speeds.
• show-cfgspace which leverages the existing kernel interfaces to read and decode from configuration space or from a file. This allows us to decode a large number of capabilities in configuration space.
• save-cfgsapce which has the ability to save configuration space out.

Here are the summaries of the commands and a number of demos:

rm@beowulf:~$ /usr/lib/pci/pcieadm show-devs -h
pcieadm: unknown option: -h

Usage:  pcieadm show-devs [-F] [-H] [-s | -o field[,...] [-p]] [filter...]

List PCI devices and functions in the system. Each <filter> selects a set
of devices to show and can be a driver name, instance, /devices path, or
b/d/f.

        -F              do not display PCI functions
        -H              omit the column header
        -o field        output fields to print
        -p              parsable output (requires -o)
        -s              list speeds and widths
rm@beowulf:~$ /usr/lib/pci/pcieadm save-cfgspace -d
pcieadm: Option -d requires an argument

Usage:  pcieadm save-cfgspace -d device output-file
        pcieadm save-cfgspace -a output-directory

Save PCI configuration space data from a device to a file or
save all devices to a specified directory.

        -a              save data from all devices
        -d device       read data from the specified device (driver instance,
                        /devices path, or b/d/f)
rm@beowulf:~$ /usr/lib/pci/pcieadm show-cfgspace -d
pcieadm: Option -d requires an argument

Usage:  pcieadm show-cfgspace [-L] [-n] [-H] -d device | -f file [filter...]
        pcieadm show-cfgspace -p -o field[,...] [-H] -d device | -f file
                              [filter...]

Print and decode PCI configuration space data from a device or file. Each
<filter> selects a given capability, sub-capability, register, or field to print.

        -d device       read data from the specified device (driver instance,
                        /devices path, or b/d/f)
        -f file         read data from the specified file
        -L              list printable fields
        -n              show printable short names
        -H              omit the column header (for -L and -p)
        -p              parsable output (requires -o)
        -o field        output fields to print (required for -p)

All of this is currently designed around using ofmt output (excepting the human readable version of reading configuration space). In addition, each of these has additional filters that can be used to design stuff. For example, here's showing all the igb devices on the system:

rm@beowulf:~$ /usr/lib/pci/pcieadm show-devs igb
BDF     TYPE           DRIVER         DEVICE
41/0/0  PCIe Gen 2x4   igb0           I350 Gigabit Network Connection
41/0/1  PCIe Gen 2x4   igb1           I350 Gigabit Network Connection

More generally here's what the speeds version of this looks like:

rm@beowulf:~$ /usr/lib/pci/pcieadm show-devs -s
BDF     DRIVER         MAXSPEED  CURSPEED  MAXWIDTH  CURWIDTH  SUPSPEEDS
0/0/0   amdzen_stub0   --        --        --        --        --
0/1/0   --             --        --        --        --        --
0/1/1   pcieb17        16.0 GT/s 8.0 GT/s  x16       x16       2.5,5.0,8.0,16.0
1/0/0   --             8.0 GT/s  8.0 GT/s  x16       x16       2.5,5.0,8.0
1/0/1   --             8.0 GT/s  8.0 GT/s  x16       x16       2.5,5.0,8.0
1/0/2   --             8.0 GT/s  8.0 GT/s  x16       x16       2.5,5.0,8.0
1/0/3   --             8.0 GT/s  8.0 GT/s  x16       x16       2.5,5.0,8.0
1/0/4   t4nex1         8.0 GT/s  8.0 GT/s  x16       x16       2.5,5.0,8.0
1/0/5   --             8.0 GT/s  8.0 GT/s  x16       x16       2.5,5.0,8.0
1/0/6   --             8.0 GT/s  8.0 GT/s  x16       x16       2.5,5.0,8.0
0/2/0   --             --        --        --        --        --
0/3/0   --             --        --        --        --        --
0/3/2   pcieb0         16.0 GT/s 8.0 GT/s  x4        x4        2.5,5.0,8.0,16.0
2/0/0   nvme0          8.0 GT/s  8.0 GT/s  x4        x4        2.5,5.0,8.0
0/4/0   --             --        --        --        --        --
0/5/0   --             --        --        --        --        --
0/7/0   --             --        --        --        --        --
0/7/1   pcieb1         16.0 GT/s 16.0 GT/s x16       x16       2.5,5.0,8.0,16.0
3/0/0   --             16.0 GT/s 16.0 GT/s x16       x16       2.5,5.0,8.0,16.0
3/0/2   --             16.0 GT/s 16.0 GT/s x16       x16       2.5,5.0,8.0,16.0
0/8/0   --             --        --        --        --        --
0/8/1   pcieb2         16.0 GT/s 16.0 GT/s x16       x16       2.5,5.0,8.0,16.0
4/0/0   --             16.0 GT/s 16.0 GT/s x16       x16       2.5,5.0,8.0,16.0
4/0/2   --             16.0 GT/s 16.0 GT/s x16       x16       2.5,5.0,8.0,16.0
4/0/3   xhci0          16.0 GT/s 16.0 GT/s x16       x16       2.5,5.0,8.0,16.0
...

Configuration space has both parsable and human output for different fields. So here's an example of seeing a single capability for igb0:

rm@beowulf:~$ pfexec /usr/lib/pci/pcieadm show-cfgspace -d igb0 pcie
PCI Express Capability (0x10)
  Capability Register: 0x2
    |--> Version: 0x2
    |--> Device/Port Type: PCIe Endpoint (0x0)
    |--> Slot Implemented: No (0x0)
    |--> Interrupt Message Number: 0x0
  Device Capabilities: 0x10008cc2
    |--> Max Payload Size Supported: 512 bytes (0x2)
    |--> Phantom Functions Supported: No (0x0)
    |--> Extended Tag Field: 5-bit (0x0)
    |--> L0s Acceptable Latency: 512 ns (0xc0)
    |--> L1 Acceptable Latency: 64 us (0xc00)
    |--> Role Based Error Reporting: supported (0x8000)
    |--> ERR_COR Subclass: unsupported (0x0)
    |--> Captured Slot Power Limit: 0x0
    |--> Captured Slot Power Limit Scale: 1.0x (0x0)
    |--> Function Level Reset: supported (0x10000000)
  Device Control: 0x2057
    |--> Correctable Error Reporting: enabled (0x1)
    |--> Non-Fatal Error Reporting: enabled (0x2)
    |--> Fatal Error Reporting: enabled (0x4)
    |--> Unsupported Request Reporting: disabled (0x0)
    |--> Relaxed Ordering: enabled (0x10)
    |--> Max Payload Size: 512 bytes (0x40)
    |--> Extended Tag Field: disabled (0x0)
    |--> Phantom Functions: disabled (0x0)
    |--> Aux Power PM: disabled (0x0)
    |--> No Snoop: disabled (0x0)
    |--> Max Read Request Size: 512 bytes (0x2000)
    |--> Bridge Configuration Retry / Function Level Reset: 0x0
  Device Status: 0x10
    |--> Correctable Error Detected: no (0x0)
    |--> Non-Fatal Error Detected: no (0x0)
    |--> Fatal Error Detected: no (0x0)
    |--> Unsupported Request Detected: no (0x0)
    |--> AUX Power Detected: yes (0x10)
    |--> Transactions Pending: no (0x0)
    |--> Emergency Power Reduction Detected: no (0x0)
  Link Capabilities: 0x42ec42
    |--> Maximum Link Speed: 5.0 GT/s (0x2)
    |--> Maximum Link Width: 0x4
    |--> ASPM Support: L0s/L1 (0xc00)
    |--> L0s Exit Latency: 2-4us (0x6000)
    |--> L1 Exit Latency: 16-32us32-64us (0x28000)
    |--> Clock Power Management: unsupported (0x0)
    |--> Surprise Down Error Reporting: unsupported (0x0)
    |--> Data Link Layer Active Reporting: unsupported (0x0)
    |--> Link Bandwidth Notification Capability: unsupported (0x0)
    |--> ASPM Optionality Compliance: compliant (0x400000)
    |--> Port Number: 0x0
  Link Control: 0x40
    |--> ASPM Control: None (0x0)
    |--> Read Completion Boundary: 64 byte (0x0)
    |--> Link Disable: not force disabled (0x0)
    |--> Retrain Link: 0x0
    |--> Common Clock Configuration: common (0x40)
    |--> Extended Sync: 0x40e0cb
    |--> Clock Power Management: 0x40e0cb
    |--> Hardware Autonomous Width: 0x40e0b9
    |--> Link Bandwidth Management Interrupt: 0x40e0cb
    |--> Link Autonomous Bandwidth Interrupt: 0x40e0cb
    |--> DRS Signaling Control: 0x40f222
  Link Status: 0x1042
    |--> Link Speed: 5.0 GT/s (0x2)
    |--> Link Width: 0x4
    |--> Link Training: no (0x0)
    |--> Slot Clock Configuration: common (0x1000)
    |--> Data Link Layer Link Active: no (0x0)
    |--> Link Bandwidth Management Status: no change (0x0)
    |--> Link Autonomous Bandwidth Status: no change (0x0)
  Slot Capabilities: 0x0
    |--> Attention Button Present: no (0x0)
    |--> Power Controller Present: no (0x0)
    |--> MRL Sensor Present: no (0x0)
    |--> Attention Indicator Present: no (0x0)
    |--> Power Indicator Present: no (0x0)
    |--> Hot-Plug Surprise: unsupported (0x0)
    |--> Hot-Plug Capable : unsupported (0x0)
    |--> Slot Power Limit Value: 0x0
    |--> Slot Power Limit Scale: 0x0
    |--> Electromechanical Interlock Present: no (0x0)
    |--> No Command Completed: 0x40e158
    |--> Physical Slot Number: 0x0
  Slot Control: 0x0
    |--> Attention Button Pressed: disabled (0x0)
    |--> Power Fault Detected: disabled (0x0)
    |--> MRL Sensor Changed: disabled (0x0)
    |--> Presence Detect Changed: disabled (0x0)
    |--> Command Complete Interrupt: disabled (0x0)
    |--> Hot Plug Interrupt Enable: disabled (0x0)
    |--> Attention Indicator Control: reserved (0x0)
    |--> Power Indicator Control: reserved (0x0)
    |--> Power Controller Control: power on (0x0)
    |--> Electromechanical Interlock Control: 0x0
    |--> Data Link Layer State Changed: disabled (0x0)
    |--> Auto Slot Power Limit: enabled (0x0)
    |--> In-Band PD: enabled (0x0)
  Slot Status: 0x0
    |--> Attention Button Pressed: no (0x0)
    |--> Power Fault Detected: no (0x0)
    |--> MRL Sensor Changed: no (0x0)
    |--> Presence Detect Changed: no (0x0)
    |--> Command Complete: no (0x0)
    |--> MRL Sensor State: closed (0x0)
    |--> Presence Detect State: not present (0x0)
    |--> Electromechanical Interlock: disengaged (0x0)
    |--> Data Link Layer State Changed: no (0x0)
  Root Control: 0x0
    |--> CRS Software Visibility: disabled (0x0)
  Root Capabilities: 0x0
    |--> System Error on Correctable Error: disabled (0x0)
    |--> System Error on Non-Fatal Error: disabled (0x0)
    |--> System Error on Fatal Error: disabled (0x0)
    |--> PME Interrupt: disabled (0x0)
    |--> CRS Software Visibility: disabled (0x0)
  Root Status: 0x0
    |--> PME Requester ID: 0x0
    |--> PME Status: deasserted (0x0)
    |--> PME Pending: no (0x0)
  Device Capabilities 2: 0x81f
    |--> Completion Timeout Ranges Supported: 0xf
      |--> 50us-10ms (0x1)
      |--> 10ms-250ms (0x2)
      |--> 250ms-4s (0x4)
      |--> 4s-64s (0x8)
    |--> Completion Timeout Disable: supported (0x10)
    |--> ARI Forwarding: unsupported (0x0)
    |--> AtomicOp Routing: unsupported (0x0)
    |--> 32-bit AtomicOp Completer: unsupported (0x0)
    |--> 64-bit AtomicOp Completer: unsupported (0x0)
    |--> 128-bit CAS Completer: unsupported (0x0)
    |--> No Ro-enabld PR-PR Passing: unsupported (0x0)
    |--> LTR Mechanism: supported (0x800)
    |--> TPH Completer: unsupported (0x0)
    |--> LN System CLS: unsupported (0x0)
    |--> 10-bit Tag Completer: unsupported (0x0)
    |--> 10-bit Tag Requester: unsupported (0x0)
    |--> OBFF: unsupported (0x0)
    |--> Extended Fmt Field Supported: unsupported (0x0)
    |--> End-End TLP Prefix Supported: unsupported (0x0)
    |--> Max End-End TLP Prefixes: 4 (0x0)
    |--> Emergency Power Reduction: unsupported (0x0)
    |--> Emergency Power Reduction Initialization Required: no (0x0)
    |--> Function Readiness Status: unsupported (0x0)
  Device Control 2: 0x0
    |--> Completion Timeout: 50us-50ms (0x0)
    |--> Completion Timeout Disabled: not disabled (0x0)
    |--> ARI Forwarding: disabled (0x0)
    |--> AtomicOp Requester: disabled (0x0)
    |--> AtomicOp Egress Blocking: unblocked (0x0)
    |--> ID-Based Ordering Request: disabled (0x0)
    |--> ID-Based Ordering Completion: disabled (0x0)
    |--> LTR Mechanism: disabled (0x0)
    |--> Emergency Power Reduction: not requested (0x0)
    |--> 10-bit Tag Requester: disabled (0x0)
    |--> OBFF: disabled (0x0)
    |--> End-End TLP Prefix Blocking: unblocked (0x0)
  Device Status 2: 0x0
  Link Capabilities 2: 0x0
    |--> Supported Link Speeds: 0x0
    |--> Crosslink: unsupported (0x0)
    |--> Lower SKP OS Generation Supported Speeds Vector: 0x0
    |--> Lower SKP OS Reception Supported Speeds Vector: 0x0
    |--> Retimer Presence Detect Supported: unsupported (0x0)
    |--> Two Retimers Presence Detect Supported: unsupported (0x0)
    |--> Device Readiness Status: unsupported (0x0)
  Link Control 2: 0x2
    |--> Target Link Speed: 5.0 GT/s (0x2)
    |--> Enter Compliance: no (0x0)
    |--> Hardware Autonomous Speed Disable: not disabled (0x0)
    |--> Selectable De-emphasis: -6 dB (0x0)
    |--> TX Margin: 0x0
    |--> Enter Modified Compliance: no (0x0)
    |--> Compliance SOS: disabled (0x0)
    |--> Compliance Preset/De-emphasis: 0x0
  Link Status 2: 0x1
    |--> Current De-emphasis Level: -3.5 dB (0x1)
    |--> Equalization 8.0 GT/s Complete: no (0x0)
    |--> Equalization 8.0 GT/s Phase 1: unsuccessful (0x0)
    |--> Equalization 8.0 GT/s Phase 2: unsuccessful (0x0)
    |--> Equalization 8.0 GT/s Phase 3: unsuccessful (0x0)
    |--> Link Equalization Request 8.0 GT/s: not requested (0x0)
    |--> Retimer Presence Detected: no (0x0)
    |--> Two Retimers Presence Detected: no (0x0)
    |--> Crosslink Resolution: unsupported (0x0)
    |--> Downstream Component Presence: link down - undetermined (0x0)
    |--> DRS Message Received: no (0x0)
  Slot Capabilities 2: 0x0
    |--> In-Band PD Disable: unsupported (0x0)
  Slot Control 2: 0x0
  Slot Status 2: 0x0

Here's an example of using parsable format for this. Critically, each field of each register can be grabbed out and the -L and -n options can help speed that up.

rm@beowulf:~$ pfexec /usr/lib/pci/pcieadm show-cfgspace -p -o short,value -d pcieb0 header1.device header1.vendor
header1.vendor:0x1022
header1.device:0x1483

I added a test suite for this that's part of the util-tests. Here's the example of that run in util-tests and on its own:

rm@beowulf:~$ pfexec /opt/util-tests/bin/utiltest 
Test: /opt/util-tests/tests/allowed-ips (run as root)             [00:00] [PASS]
Test: /opt/util-tests/tests/chown_test (run as root)              [00:00] [PASS]
Test: /opt/util-tests/tests/date_test (run as root)               [00:00] [PASS]
Test: /opt/util-tests/tests/find/findtest (run as root)           [00:00] [PASS]
Test: /opt/util-tests/tests/grep_test (run as root)               [00:03] [PASS]
Test: /opt/util-tests/tests/head/head_test (run as root)          [00:00] [PASS]
Test: /opt/util-tests/tests/libjedec_test (run as root)           [00:00] [PASS]
Test: /opt/util-tests/tests/libsff/libsff (run as root)           [00:00] [PASS]
Test: /opt/util-tests/tests/make_test (run as root)               [00:00] [PASS]
Test: /opt/util-tests/tests/mdb/mdbtest (run as root)             [00:00] [PASS]
Test: /opt/util-tests/tests/mergeq/mqt (run as root)              [00:00] [PASS]
Test: /opt/util-tests/tests/mergeq/wqt (run as root)              [00:00] [PASS]
Test: /opt/util-tests/tests/pcidbtest (run as root)               [00:01] [PASS]
Test: /opt/util-tests/tests/pcieadmtest (run as root)             [00:03] [PASS]
Test: /opt/util-tests/tests/printf_test (run as root)             [00:00] [PASS]
Test: /opt/util-tests/tests/set-linkprop (run as root)            [00:00] [PASS]
Test: /opt/util-tests/tests/sleep/sleeptest (run as root)         [00:23] [PASS]
Test: /opt/util-tests/tests/smbios (run as root)                  [00:00] [PASS]
Test: /opt/util-tests/tests/xargs_test (run as root)              [00:00] [PASS]
Test: /opt/util-tests/tests/awk/runtests.sh (run as nobody)       [02:54] [PASS]
Test: /opt/util-tests/tests/ctf/precheck (run as root)            [00:00] [PASS]
Test: /opt/util-tests/tests/ctf/ctftest (run as root)             [00:06] [PASS]
Test: /opt/util-tests/tests/demangle/afl-fast (run as root)       [00:01] [PASS]
Test: /opt/util-tests/tests/demangle/gcc-libstdc++ (run as root)  [00:00] [PASS]
Test: /opt/util-tests/tests/demangle/llvm-stdcxxabi (run as root) [00:00] [PASS]
Test: /opt/util-tests/tests/libcustr/custr_remove (run as root)   [00:00] [PASS]
Test: /opt/util-tests/tests/libcustr/custr_trunc (run as root)    [00:00] [PASS]
Test: /opt/util-tests/tests/libnvpair_json/json_00_blank (run as root) [00:00] [PASS]
Test: /opt/util-tests/tests/libnvpair_json/json_01_boolean (run as root) [00:00] [PASS]
Test: /opt/util-tests/tests/libnvpair_json/json_02_numbers (run as root) [00:00] [PASS]
Test: /opt/util-tests/tests/libnvpair_json/json_03_empty_arrays (run as root) [00:00] [PASS]
Test: /opt/util-tests/tests/libnvpair_json/json_04_number_arrays (run as root) [00:00] [PASS]
Test: /opt/util-tests/tests/libnvpair_json/json_05_strings (run as root) [00:00] [PASS]
Test: /opt/util-tests/tests/libnvpair_json/json_06_nested (run as root) [00:00] [PASS]
Test: /opt/util-tests/tests/libnvpair_json/json_07_nested_arrays (run as root) [00:00] [PASS]
Test: /opt/util-tests/tests/sed/sed_addr (run as root)            [00:00] [PASS]
Test: /opt/util-tests/tests/sed/multi_test (run as root)          [00:00] [PASS]

Results Summary
PASS      37

Running Time:   00:03:41
Percent passed: 100.0%
Log directory:  /var/tmp/test_results/20210330T151449
rm@beowulf:~$ /opt/util-tests/tests/pcieadmtest
TEST PASSED: invalid arguments 
TEST PASSED: invalid arguments -d
TEST PASSED: invalid arguments foobar
TEST PASSED: invalid arguments save-cfgspace
TEST PASSED: invalid arguments save-cfgspace -a
TEST PASSED: invalid arguments save-cfgspace -d
TEST PASSED: invalid arguments save-cfgspace -d final
TEST PASSED: invalid arguments save-cfgspace -a -d fantasy
TEST PASSED: invalid arguments show-devs -h
TEST PASSED: invalid arguments show-devs -p
TEST PASSED: invalid arguments show-devs -s -o
TEST PASSED: invalid arguments show-cfgspace
TEST PASSED: invalid arguments show-cfgspace -d -H
TEST PASSED: invalid arguments show-cfgspace -d
TEST PASSED: invalid arguments show-cfgspace -f
TEST PASSED: invalid arguments show-cfgspace -h
TEST PASSED: invalid arguments show-cfgspace -L
TEST PASSED: invalid arguments show-cfgspace -L -n -f /opt/util-tests/tests/pci/igb.pci
TEST PASSED: invalid arguments show-cfgspace -L -p -f /opt/util-tests/tests/pci/igb.pci
TEST PASSED: invalid arguments show-cfgspace -p -f /opt/util-tests/tests/pci/igb.pci
TEST PASSED: invalid arguments show-cfgspace -o foo -f /opt/util-tests/tests/pci/igb.pci
TEST PASSED: invalid arguments show-cfgspace -L -o foo -f /opt/util-tests/tests/pci/igb.pci
TEST PASSED: show-cfgspace -f /dev/stdin header0.vendor header0.device
TEST PASSED: show-cfgspace -L -f /dev/stdin header0.vendor header0.device
TEST PASSED: show-cfgspace -n -f /dev/stdin header0.vendor header0.device
TEST PASSED: show-cfgspace -L -H -f /dev/stdin header0.vendor header0.device
TEST PASSED: show-cfgspace -f /dev/stdin ltr
TEST PASSED: show-cfgspace -p -o short,value -f /dev/stdin ltr
TEST PASSED: show-cfgspace -p -o short,value -f /dev/stdin header0.vendor header0.device
TEST PASSED: show-cfgspace -f /dev/stdin ht
TEST PASSED: show-cfgspace -f /dev/stdin ht.msi
TEST PASSED: show-cfgspace -f /dev/stdin ht.msi.command
TEST PASSED: show-cfgspace -p -o value,short -f /dev/stdin ht
TEST PASSED: show-cfgspace -p -o value,short -f /dev/stdin ht.msi
TEST PASSED: show-cfgspace -p -o value,short -f /dev/stdin ht.msi.command
TEST PASSED: show-cfgspace -f /dev/stdin pcie.linksts atelier
TEST PASSED: show-cfgspace -p -o short,value -f /dev/stdin pcie.linksts atelier
All tests passed successfully!

In addition, thanks to folks in the community, I've run this against a couple hundred different devices and added support for a fair number of existing capabilities that we've found in the field.