Bug #8702

PCI addresses with physaddr > 0xffffffff can't be mapped in

Added by Robert Mustacchi 4 months ago. Updated about 2 months ago.

Status:ClosedStart date:2017-10-04
Priority:NormalDue date:
Assignee:Robert Mustacchi% Done:

100%

Category:driver - device drivers
Target version:-
Difficulty:Medium Tags:

Description

Enabling the BIOS option to enable decoding addresses above 4GB for PCIe devices causes ixgbe to fail attach due to:

2012-10-18T00:17:16.922070+00:00 headnode ixgbe: [ID 611667 kern.warning] WARNING: ixgbe0: Failed to map device registers
2012-10-18T00:17:16.922073+00:00 headnode ixgbe: [ID 611667 kern.warning] WARNING: ixgbe1: Failed to map device registers

The assigned address is indeed above 4GB. Disabling this option eliminates the problem.

--

Unfortunately the npe nexus driver does not support this. See the following comment in npe_bus_map:

 648         case PCI_ADDR_MEM64:
 649                 /*
 650                  * We can't handle 64-bit devices that are mapped above
 651                  * 4G or that are larger than 4G.
 652                  */
 653                 if (pci_rp->pci_phys_mid != 0 || pci_rp->pci_size_hi != 0)
 654                         return (DDI_FAILURE);

A bit more background. From digging into this, the x86 root nexus driver has never supported 64-bit addresses. In particular the nexus driver uses a struct regspec, which only allow for 32-bit quantities to be encoded. The interface between the root nexus driver and its parent driver are private. To deal with not needing to modify a bunch of other nexus drivers, we've used another flag that the PCI and NPE drivers use.


Related issues

Related to illumos gate - Feature #8703: pci/npe DDI_CTLOPS_REGSIZE should be 64-bit aware Closed 2017-10-04
Related to illumos gate - Bug #8704: want OFF_MAX in the kernel Closed 2017-10-04
Related to illumos gate - Bug #8705: unsupported 64-bit prefetch memory on pci-pci bridge Closed 2017-10-04

History

#1 Updated by Robert Mustacchi 4 months ago

  • Related to Feature #8703: pci/npe DDI_CTLOPS_REGSIZE should be 64-bit aware added

#2 Updated by Robert Mustacchi 4 months ago

  • Related to Bug #8704: want OFF_MAX in the kernel added

#3 Updated by Robert Mustacchi 4 months ago

  • Related to Bug #8705: unsupported 64-bit prefetch memory on pci-pci bridge added

#4 Updated by Electric Monk about 2 months ago

  • Status changed from New to Closed

git commit 1f0c5e612d8dfd7d1a8f4487cef753357b9909f5

commit  1f0c5e612d8dfd7d1a8f4487cef753357b9909f5
Author: Robert Mustacchi <rm@joyent.com>
Date:   2017-12-04T17:23:22.000Z

    8702 PCI addresses with physaddr > 0xffffffff can't be mapped in
    8703 pci/npe DDI_CTLOPS_REGSIZE should be 64-bit aware
    8704 want OFF_MAX in the kernel
    8705 unsupported 64-bit prefetch memory on pci-pci bridge
    Reviewed by: Patrick Mooney <patrick.mooney@joyent.com>
    Reviewed by: Jerry Jelinek <jerry.jelinek@joyent.com>
    Reviewed by: Albert Lee <trisk@forkgnu.org>
    Approved by: Gordon Ross <gwr@nexenta.com>

Also available in: Atom