illumos gate

bhyve's passthrough feature sometimes needs to mmap() the BAR containing the MSI-X table due to the way it emulates that. It does this via a mmap() of the passthrough device /dev/pptN.

Internally, the ppt driver calls devmap_devmem_setup() but it assumes that BAR X equates to DDI register X+1, which is not always the case if there are any 64-bit BARs (since they use two slots) or any unused BARs.

        if ((bar = ppt_find_msix_table_bar(ppt)) == -1)
                return (EINVAL);

        /*
         * Add 1 to the BAR number to get the register number used by DDI.
         * Register 0 corresponds to PCI config space, the PCI BARs start at 1.
         */
        bar += 1;

        err = devmap_devmem_setup(dhp, ppt->pptd_dip, NULL, bar, off, len,
            PROT_USER | PROT_READ | PROT_WRITE, IOMEM_DATA_CACHED, &ppt_attr);

For example, consider this igb device which is bound to the ppt driver:

The MSI-X table is in BAR 4:

theeo# pcieadm show-cfgspace -d ppt0 msix.table.bir
MSI-X Capability (0x11)
  Table Offset: 0x4
    |--> Table BIR: BAR 4 (0x4)

theeo# pcieadm show-cfgspace -d ppt0 header0.bar4
Device ppt0 -- Type 0 Header
  Base Address Register 4
    |--> Space: Memory Space (0x0)
    |--> Address: 0xdf3e0000
    |--> Address: 64-bit (0x2)
    |--> Prefetchable: no (0x0)

ppt has the correct information for this:

theeo# mdb -k
> ::prtconf -d ppt -i 0 | ::devinfo -d | ::print struct pptdev pptd_bars[4]
pptd_bars[4] = {
   pptd_bars[4].base = 0xdf3e0000
   pptd_bars[4].size = 0x4000
   pptd_bars[4].type = 0x3000000
   pptd_bars[4].io_handle = 0
   pptd_bars[4].io_ptr = 0
}

However, the assumption that it is ddi register 5 which should be mapped is incorrect since BAR1 is part of the 64-bit BAR0 and BAR3 is empty. It should be mapped as register 3.

theeo# prtconf -v /dev/ppt0
pci15d9,1f41, instance #0
        name='assigned-addresses' type=int items=15
            value=
              8300a310.00000000.df300000.00000000.00020000
              8100a318.00000000.0000e060.00000000.00000020
              8300a320.00000000.df3e0000.00000000.00004000
        name='reg' type=int items=20
            value=
              0000a300.00000000.00000000.00000000.00000000
              0300a310.00000000.00000000.00000000.00020000
              0100a318.00000000.00000000.00000000.00000020
              0300a320.00000000.00000000.00000000.00004000
  Base Address Register 0
    |--> Space: Memory Space (0x0)
    |--> Address: 0xdf300000
    |--> Address: 64-bit (0x2)
    |--> Prefetchable: no (0x0)
  Base Address Register 2
    |--> Space: I/O Space (0x1)
    |--> Address: 0xe060
  Base Address Register 3
    |--> Space: Memory Space (0x0)
    |--> Address: 0x0
    |--> Address: 32-bit (0x0)
    |--> Prefetchable: no (0x0)
  Base Address Register 4
    |--> Space: Memory Space (0x0)
    |--> Address: 0xdf3e0000
    |--> Address: 64-bit (0x2)
    |--> Prefetchable: no (0x0)

The BAR map in ppt has the gaps so it should be possible to derive this properly.

> ::prtconf -d ppt -i 0 | ::devinfo -d | ::print struct pptdev pptd_bars
pptd_bars = [
    {
        base = 0xdf300000
        size = 0x20000
        type = 0x3000000
        io_handle = 0
        io_ptr = 0
    },
    {
        base = 0
        size = 0
        type = 0
        io_handle = 0
        io_ptr = 0
    },
    {
        base = 0xe060
        size = 0x20
        type = 0x1000000
        io_handle = 0xfffffe2dba749640
        io_ptr = 0xe060
    },
    {
        base = 0
        size = 0
        type = 0
        io_handle = 0
        io_ptr = 0
    },
    {
        base = 0xdf3e0000
        size = 0x4000
        type = 0x3000000
        io_handle = 0
        io_ptr = 0
    },
    {
        base = 0
        size = 0
        type = 0
        io_handle = 0
        io_ptr = 0
    },
]