Writes to MSI-X table should be DWords, not QWords
Accesses to MSI-X Table entries should be 32-bits (DWords) at a time. However, we use QWords to access the high and low portions of the address in the MSI-X table. This causes problems with some platforms, which manifests as an inability to configure some or all interrupts on the system.
This description from our internal bug tracking system details the problem further. Credit goes to Tycho Nightingale for figuring this out.
The folks that developed the Intel Communications Chipset 89xx are probably as confused as I am by the apparent ambiguity of the PCI specification regarding the MSI-X interrupt table. In one part of the spec you read: "To request service using a given MSI-X Table entry, a function performs a DWORD memory write transaction using the contents of the Message Data field entry for data, the contents of the Message Upper Address field for the upper 32 bits of address, and the contents of the Message Address field entry for the lower 32 bits of address. A memory read transaction from the address targeted by the MSI-X message produces undefined results." and then later on: "For all accesses to MSI-X Table and MSI-X PBA fields, software must use aligned full DWORD or aligned full QWORD transaction." so one is left wondering: are Qword accesses are okay? At least the 89xx'ers made it clear that for their device "software can access Dword entities." Too bad that our OS assumed Qwords were okay and had good luck with that until it met the NIC on the Quanta box. To get MSI-X interrupts working on that platform, I converted the single Qword accesses into double Dwords; It's off the critical path, so the performance impact is irrelevant, and furthermore now our behavior is same as what I found in Linux and FreeBSD.