Project

General

Profile

Actions

Bug #14373

closed

ppt driver maps BAR to wrong ddi register

Added by Andy Fiddaman 5 months ago. Updated 4 months ago.

Status:
Closed
Priority:
Normal
Assignee:
Category:
bhyve
Start date:
Due date:
% Done:

100%

Estimated time:
Difficulty:
Medium
Tags:
Gerrit CR:

Description

bhyve's passthrough feature sometimes needs to mmap() the BAR containing the MSI-X table due to the way it emulates that. It does this via a mmap() of the passthrough device /dev/pptN.

Internally, the ppt driver calls devmap_devmem_setup() but it assumes that BAR X equates to DDI register X+1, which is not always the case if there are any 64-bit BARs (since they use two slots) or any unused BARs.

        if ((bar = ppt_find_msix_table_bar(ppt)) == -1)
                return (EINVAL);

        /*
         * Add 1 to the BAR number to get the register number used by DDI.
         * Register 0 corresponds to PCI config space, the PCI BARs start at 1.
         */
        bar += 1;

        err = devmap_devmem_setup(dhp, ppt->pptd_dip, NULL, bar, off, len,
            PROT_USER | PROT_READ | PROT_WRITE, IOMEM_DATA_CACHED, &ppt_attr);

For example, consider this igb device which is bound to the ppt driver:

The MSI-X table is in BAR 4:

theeo# pcieadm show-cfgspace -d ppt0 msix.table.bir
MSI-X Capability (0x11)
  Table Offset: 0x4
    |--> Table BIR: BAR 4 (0x4)

theeo# pcieadm show-cfgspace -d ppt0 header0.bar4
Device ppt0 -- Type 0 Header
  Base Address Register 4
    |--> Space: Memory Space (0x0)
    |--> Address: 0xdf3e0000
    |--> Address: 64-bit (0x2)
    |--> Prefetchable: no (0x0)

ppt has the correct information for this:

theeo# mdb -k
> ::prtconf -d ppt -i 0 | ::devinfo -d | ::print struct pptdev pptd_bars[4]
pptd_bars[4] = {
   pptd_bars[4].base = 0xdf3e0000
   pptd_bars[4].size = 0x4000
   pptd_bars[4].type = 0x3000000
   pptd_bars[4].io_handle = 0
   pptd_bars[4].io_ptr = 0
}

However, the assumption that it is ddi register 5 which should be mapped is incorrect since BAR1 is part of the 64-bit BAR0 and BAR3 is empty. It should be mapped as register 3.

theeo# prtconf -v /dev/ppt0
pci15d9,1f41, instance #0
        name='assigned-addresses' type=int items=15
            value=
              8300a310.00000000.df300000.00000000.00020000
              8100a318.00000000.0000e060.00000000.00000020
              8300a320.00000000.df3e0000.00000000.00004000
        name='reg' type=int items=20
            value=
              0000a300.00000000.00000000.00000000.00000000
              0300a310.00000000.00000000.00000000.00020000
              0100a318.00000000.00000000.00000000.00000020
              0300a320.00000000.00000000.00000000.00004000
  Base Address Register 0
    |--> Space: Memory Space (0x0)
    |--> Address: 0xdf300000
    |--> Address: 64-bit (0x2)
    |--> Prefetchable: no (0x0)
  Base Address Register 2
    |--> Space: I/O Space (0x1)
    |--> Address: 0xe060
  Base Address Register 3
    |--> Space: Memory Space (0x0)
    |--> Address: 0x0
    |--> Address: 32-bit (0x0)
    |--> Prefetchable: no (0x0)
  Base Address Register 4
    |--> Space: Memory Space (0x0)
    |--> Address: 0xdf3e0000
    |--> Address: 64-bit (0x2)
    |--> Prefetchable: no (0x0)

The BAR map in ppt has the gaps so it should be possible to derive this properly.

> ::prtconf -d ppt -i 0 | ::devinfo -d | ::print struct pptdev pptd_bars
pptd_bars = [
    {
        base = 0xdf300000
        size = 0x20000
        type = 0x3000000
        io_handle = 0
        io_ptr = 0
    },
    {
        base = 0
        size = 0
        type = 0
        io_handle = 0
        io_ptr = 0
    },
    {
        base = 0xe060
        size = 0x20
        type = 0x1000000
        io_handle = 0xfffffe2dba749640
        io_ptr = 0xe060
    },
    {
        base = 0
        size = 0
        type = 0
        io_handle = 0
        io_ptr = 0
    },
    {
        base = 0xdf3e0000
        size = 0x4000
        type = 0x3000000
        io_handle = 0
        io_ptr = 0
    },
    {
        base = 0
        size = 0
        type = 0
        io_handle = 0
        io_ptr = 0
    },
]

Related issues

Blocks illumos gate - Feature #14372: bhyve upstream sync 2022 JanuaryClosedAndy Fiddaman

Actions
Actions #1

Updated by Andy Fiddaman 5 months ago

Actions #2

Updated by Andy Fiddaman 5 months ago

  • Subject changed from ppt driver maps BAR to wrong register to ppt driver maps BAR to wrong ddi register
Actions #3

Updated by Andy Fiddaman 5 months ago

  • Description updated (diff)
Actions #4

Updated by Electric Monk 5 months ago

  • Gerrit CR set to 1920
Actions #5

Updated by Andy Fiddaman 5 months ago

  • Description updated (diff)
Actions #6

Updated by Andy Fiddaman 5 months ago

  • Description updated (diff)
Actions #7

Updated by Andy Fiddaman 4 months ago

I've tested this with the updated bhyve sync bits, and with a small test program I wrote to open a /dev/pptX device and mmap() the descriptor. Test devices with the MSI-X table in bar0 worked before and after, those where it was not in bar0 and there were gaps in the bar list (due to 64-bit or unused bars occurring before the MSI-X table) failed before and worked afterwards. Jorge has also verified that this patch fixes his VMs using passthrough with the new bhyve sync update - the new bhyve unconditionally maps the MSI-X BAR whereas the old only mapped it in certain circumstances.

Actions #8

Updated by Electric Monk 4 months ago

  • Status changed from In Progress to Closed
  • % Done changed from 0 to 100

git commit 7ca35597a3c3894e7222816a3f7ed2be2a0686e8

commit  7ca35597a3c3894e7222816a3f7ed2be2a0686e8
Author: Andy Fiddaman <omnios@citrus-it.co.uk>
Date:   2022-01-11T20:00:13.000Z

    14373 ppt driver maps BAR to wrong ddi register
    Reviewed by: Igor Kozhukhov <igor@dilos.org>
    Reviewed by: Jorge Schrauwen <registration@blackdot.be>
    Reviewed by: Hans Rosenfeld <rosenfeld@grumpf.hope-2000.org>
    Approved by: Dan McDonald <danmcd@joyent.com>

Actions

Also available in: Atom PDF