Project

General

Profile

Feature #11356

Want Fortville TSO support

Added by Robert Mustacchi 3 months ago. Updated 3 months ago.

Status:
Closed
Priority:
Normal
Assignee:
Category:
driver - device drivers
Start date:
Due date:
% Done:

100%

Estimated time:
Difficulty:
Medium
Tags:

Description

We should add TCP Segmentation Offloading (aka TSO, aka LSO) support to the i40e driver.

Testing

Test configuration 1

The changes were tested using a pair of J3301 nodes in SPC staging:

system 1
CSN: 8S8PRD2
NIC: Intel XXV710 for 25GbE SFP28
running smartos PI with OS-5225 changes (v1.0.2 i40e driver)

system 2
CSN: 8SBLRD2
NIC: Intel XXV710 for 25GbE SFP28
running stock smartos PI (v1.0.1 i40e driver)

The i40e interfaces on both systems are connected to a common network via 25gbs links.
iperf network testing scenarios

iperf testing will be performed using the following network parameters:

scenario 1: LSO disabled on system 1, 1500 MTU (regression)

scenario 2: LSO enabled on system 1, 1500 MTU
results: see attachments v101_1500mtu_25g_iperf.txt and v102_1500mtu_25g_iperf.txt for precise numbers - but in summary, the average throughput for the existing v1.0.1 driver was 7.93gps and the average throughput for the new v1.0.2 driver was 10.1gbps

scenario 3: LSO enabled on system 1, on both systems, two i40e interfaces combined via LACP, 1500 MTU
results: see attachments v101_1500mtu_25g_aggr_iperf.txt and v102_1500mtu_25g_aggr_iperf.txt for iperf output - but in summary, the average throughput for the existing v1.0.1 driver was 19.9gps and the average throughput for the new v1.0.2 driver was roughly 25.8gbps

For each of the above test scenarios, iperf testing was done first with system 2 as the sender (to establish a baseline using the stock i40e driver), and then with system 1 as the sender.

Memory Leak Testing

To test for leaks in the i40e driver, network load (driven by iperf) was run between system 1 (running with kmem_flags=0xf) to system 2 for several hours. Afterwards, I forced a crash dump on system 1 and ran "findleaks" on the dump. The only reported leaks were in the mpt_sas driver. I filed OS-6900 to track those leaks.

Test Configuration 2

In this test configuration, the i40e interface is connected back-to-back to a 10g interface on a separate system (no switch). This configuration was used to sanity test TX from i40e using jumbo frames.

system1
SMCI,SSG-2029P-ACR24L
onboard Intel X722 for 10GBASE-T

system2

SMCI,SSG-2028R-ACR24L

onboard Intel X540-AT2

scenario 1: LSO disabled on system 1, 9000 MTU

For test scenario 1 above, iperf was first on stock smartos bits (i.e. the existing v1.0.1 i40e driver), to establish a baseline. It was then run again with smartos bits that include the changes for LSO. (i.e. the v1.0.2 i40e driver).

Results: see v101_9000mtu.txt and v102_LSO_disabled_9000mtu.txt for captured iperf output, but the performance observed was almost identical between the two configurations, which was the expected result.

scenario 2: LSO enabled on system 1, 9000 MTU

Results: see v102_LSO_enabled_9000mtu.txt - results similar to above and system was stable.


Files

v102_1500mtu_25g_iperf.txt (30.5 KB) v102_1500mtu_25g_iperf.txt Robert Mustacchi, 2019-07-08 05:26 PM
v102_1500mtu_25g_aggr_iperf.txt (329 KB) v102_1500mtu_25g_aggr_iperf.txt Robert Mustacchi, 2019-07-08 05:26 PM
v102_LSO_disabled_9000mtu.txt (32.2 KB) v102_LSO_disabled_9000mtu.txt Robert Mustacchi, 2019-07-08 05:26 PM
v102_LSO_enabled_9000mtu.txt (31.1 KB) v102_LSO_enabled_9000mtu.txt Robert Mustacchi, 2019-07-08 05:26 PM
v101_1500mtu_25g_iperf.txt (30.6 KB) v101_1500mtu_25g_iperf.txt Robert Mustacchi, 2019-07-08 05:26 PM
v101_1500mtu_25g_aggr_iperf.txt (329 KB) v101_1500mtu_25g_aggr_iperf.txt Robert Mustacchi, 2019-07-08 05:26 PM
v101_9000mtu.txt (31.1 KB) v101_9000mtu.txt Robert Mustacchi, 2019-07-08 05:26 PM

History

#1

Updated by Robert Mustacchi 3 months ago

This also includes the fixes for OS-7082, OS-7086, OS-7101.

#2

Updated by Electric Monk 3 months ago

  • Status changed from New to Closed

git commit 09aee6126f680324a9b019f9b4c77309dc611bf9

commit  09aee6126f680324a9b019f9b4c77309dc611bf9
Author: Ryan Zezeski <rpz@joyent.com>
Date:   2019-07-15T18:17:00.000Z

    11356 Want Fortville TSO support
    11357 want i40e multi-group support
    11358 i40e_alloc_ring_mem() unwinds when it shouldn't
    11359 Rework i40e transmit descriptor logic
    Portions contributed by: Rob Johnston <rob.johnston@joyent.com>
    Reviewed by: Robert Mustacchi <rm@joyent.com>
    Reviewed by: Patrick Mooney <patrick.mooney@joyent.com>
    Reviewed by: Randy Fishel <randyf@sibernet.com>
    Approved by: Garrett D'Amore <garrett@damore.org>

Also available in: Atom PDF