Project

General

Profile

Bug #4217

Writes to MSI-X table should be DWords, not QWords

Added by Garrett D'Amore about 7 years ago. Updated about 7 years ago.

Status:
Resolved
Priority:
High
Category:
kernel
Start date:
2013-10-16
Due date:
% Done:

100%

Estimated time:
Difficulty:
Medium
Tags:
Gerrit CR:

Description

Accesses to MSI-X Table entries should be 32-bits (DWords) at a time. However, we use QWords to access the high and low portions of the address in the MSI-X table. This causes problems with some platforms, which manifests as an inability to configure some or all interrupts on the system.

This description from our internal bug tracking system details the problem further. Credit goes to Tycho Nightingale for figuring this out.

The folks that developed the Intel Communications Chipset 89xx are probably as confused as I am by the apparent ambiguity of the PCI specification regarding the MSI-X interrupt table.  In one part of the spec you read:

    "To request service using a given MSI-X Table entry, a function performs a DWORD memory write transaction using the contents of the Message Data field entry for data, the contents of the Message Upper Address field for the upper 32 bits of address, and the contents of the Message Address field entry for the lower 32 bits of address. A memory read transaction from the address targeted by the MSI-X message produces undefined results." 

and then later on:

    "For all accesses to MSI-X Table and MSI-X PBA fields, software must use aligned full DWORD or aligned full QWORD transaction." 

so one is left wondering: are Qword accesses are okay?

At least the 89xx'ers made it clear that for their device "software can access Dword entities."  Too bad that our OS assumed Qwords were okay and had good luck with that until it met the NIC on the Quanta box.

To get MSI-X interrupts working on that platform, I converted the single Qword accesses into double Dwords;  It's off the critical path, so the performance impact is irrelevant, and furthermore now our behavior is same as what I found in Linux and FreeBSD.
#1

Updated by Garrett D'Amore about 7 years ago

That description got bungled by the tracking system:

The folks that developed the Intel Communications Chipset 89xx are probably as confused as I am by the apparent ambiguity of the PCI specification regarding the MSI-X interrupt table. In one part of the spec you read:

"To request service using a given MSI-X Table entry, a function performs a DWORD memory write transaction using the contents of the Message Data field entry for data, the contents of the Message Upper Address field for the upper 32 bits of address, and the contents of the Message Address field entry for the lower 32 bits of address. A memory read transaction from the address targeted by the MSI-X message produces undefined results."

and then later on:

"For all accesses to MSI-X Table and MSI-X PBA fields, software must use aligned full DWORD or aligned full QWORD transaction."

so one is left wondering: are Qword accesses are okay?

At least the 89xx'ers made it clear that for their device "software can access Dword entities." Too bad that our OS assumed Qwords were okay and had good luck with that until it met the NIC on the Quanta box.

To get MSI-X interrupts working on that platform, I converted the single Qword accesses into double Dwords; It's off the critical path, so the performance impact is irrelevant, and furthermore now our behavior is same as what I found in Linux and FreeBSD.

#2

Updated by Marcel Telka about 7 years ago

Garrett D'Amore wrote:

That description got bungled by the tracking system:

The Description is editable. I just fixed it for you.

#3

Updated by Garrett D'Amore about 7 years ago

  • Status changed from In Progress to Resolved
  • % Done changed from 60 to 100
  • Tags deleted (needs-triage)

commit 8181b438236881e6d31f7155101c85c011a9b6bb
Author: Garrett D'Amore <garrett@openindiana.(none)>
Date: Wed Oct 16 14:32:48 2013 -0700

4217 Writes to MSI-X table should be DWords, not QWords
Reviewed by: Robert Mustacchi &lt;&gt;
Reviewed by: Steven Hartland &lt;&gt;
Reviewed by: Marcel Telka &lt;&gt;
Approved by: Dan McDonald &lt;&gt;

Also available in: Atom PDF