mlxcx command interface should allow concurrent commands and be interrupt driven
At the moment only a single command can be submitted to the HCA at a time, and then that command is polled for completion.
The HCA supports up to 32 concurrent commands with interrupt driven completions, we should move to that.Alongside this, the method for acquiring pages for the hardware had to be slightly modified.
- The number of pages in a single command was increased from 512 to 4096.
- Previously, when the hardware required pages, the driver would iterate in a loop giving pages in 512 page chunks at a time, inside the interrupt thread. When this was moved to a taskq, it unblocked the interrupt and caused a storm of page request interrupts as each 512 did not satisfy the hardware's request. The solution is to remove the iteration, and just give pages to the hardware in chunks of 4096, and let the h/w drive another interrupt until its needs have been satisfied.
Updated by Paul Winder 2 months ago
When Robert released his work-in-progress in 2019, I started investigating and this is one of the first changes I made.
Since then Alex finished off the driver, and again at Racktop I pulled in the changes I had made and have been running with them since we started using the driver in production.
Updated by Paul Winder 25 days ago
- The changes are from Racktop's production version of the driver, and as such have been through Racktop QA.
- The command interface in particular is stressed during driver load and unload. Changes tested by repeated loading and unloading driver.
- Basic functionality tested using iperf to provide a load, created and used aggrs, vnics, changed MTUs.
Updated by Electric Monk 25 days ago
- Status changed from In Progress to Closed
- % Done changed from 80 to 100
commit 5f0e3176f407dfb9d989b5dcc94a6d5384d0b142 Author: Paul Winder <email@example.com> Date: 2020-07-10T15:41:30.000Z 12797 mlxcx max flow table limit can be exceeded 12798 mlxcx command interface should allow concurrent commands and be interrupt driven 12799 mlxcx #if defined for MAC_VLAN_UNTAGGED is redundant 12890 mlxcx uses excessive stack space causing stack overflow panic Reviewed by: Garrett D'Amore <firstname.lastname@example.org> Reviewed by: Andy Stormont <email@example.com> Reviewed by: Robert Mustacchi <firstname.lastname@example.org> Reviewed by: Alex Wilson <email@example.com> Approved by: Dan McDonald <firstname.lastname@example.org>