flowcontrol set to 'bi' loses IP connectivity on ixgbe driven X520
With flowcontrol set to no, tx or rx on an X520 interface driven by ixgbe connectivity seems fine however setting it to 'bi' causes a loss of IP connectivity. Seeing this at Nexenta with both NS3 & NS4
root@host1# dladm set-linkprop -p flowctrl=bi ixgbe0 root@host1# dladm show-linkprop -p flowctrl ixgbe0 LINK PROPERTY PERM VALUE DEFAULT POSSIBLE ixgbe0 flowctrl rw bi no no,tx,rx,bi root@host1# ping -s 192.0.2.31 ( ping fails with 100% packet loss ) root@host2# dladm set-linkprop -p flowctrl=bi ixgbe0 root@host2# dladm show-linkprop -p flowctrl ixgbe0 LINK PROPERTY PERM VALUE DEFAULT POSSIBLE ixgbe0 flowctrl rw bi no no,tx,rx,bi root@host2# ping -s 192.0.2.30 ( ping fails with 100% packet loss )
Updated by Kevin Crowe over 7 years ago
I have a large changeset compliments of Joerg Goltermann that I've trimmed down to just the flow control changes that fixes this problem. Initial tests on X540's show that flowcontrol isn't a problem however X520's I've tested on show this problem with flowcontrol=bi and with the pared down flowcontrol changes the problem no longer exists.
Tested on this X520 based HW (as well as X540's and at least one other X520 based card):
root@host1:/export/home/admin# echo "::prtconf" | mdb -k | grep ixgbe
ffffff090641cab0 pciex8086,10fb, instance #0 (driver name: ixgbe)
ffffff090641c828 pciex8086,10fb, instance #1 (driver name: ixgbe)
Updated by Kevin Crowe about 7 years ago
Tested changes on a variety of X540 & X520 interfaces, specifically the breakage only exists on X520's as follows:
Using the current Illumos ixgbe driver on X520's to repro the problem flow contrl set to "bi" breaks (note that it breaks with just pings too but I was using iperf to check performance as well as connectivity at the time):
root@host-a:~# dladm set-linkprop -p flowctrl=bi ixgbe0 root@host-a:~# dladm show-linkprop -p flowctrl ixgbe0 LINK PROPERTY PERM VALUE DEFAULT POSSIBLE ixgbe0 flowctrl rw bi no no,tx,rx,bi root@host-a:~# ./iperf -c 192.0.2.31 -P 4 [...no output here...just hangs...] root@host-b:~# dladm set-linkprop -p flowctrl=bi ixgbe0 root@host-b:~# dladm show-linkprop -p flowctrl ixgbe0 LINK PROPERTY PERM VALUE DEFAULT POSSIBLE ixgbe0 flowctrl rw bi no no,tx,rx,bi root@host-b:~# ./iperf -s ------------------------------------------------------------ Server listening on TCP port 5001 TCP window size: 128 KByte (default) ------------------------------------------------------------ [...no output here...just hangs...]
Set it back to "no" and ip connectivity resumes.
Putting the ixgbe driver with Joerg's pared down flow control and hi-water changes in place and rebooting and all 4 flow control settings (no,tx,rx,bi) work fine with no loss of IP connectivity.
Updated by Kevin Crowe about 7 years ago
Some info on the source of the problem and justification for the changes:
One part of the fix for setting flow control is these lines in ixgbe_gld.c in the function ixgbe_m_setprop().
This is one of the functions registered with MAC callbacks thus it gets called in
response to MAC layer properties getting set ( "dladm set-linkprop" and such
but that's about the extent of my experience in that code so...)
/* Don't autoneg if forcing a value */ ixgbe->hw.fc.disable_fc_autoneg = TRUE; (void) ixgbe_fc_enable(hw);
Basically setting that to TRUE tells the next function we call:
ixgbe_fc_enable() not to negociate FC but rather set it as the user
requested. Now, when changing FC on the fly it gets set and works
properly as the user requested.
Previously the code in ixgbe_m_setprop() never even called ixgbe_fc_enable()
so now if the flow control settings are set it sets "disable flowcontrol
autoneg" to true (i.e. don't autonegociate flowcontrol) and it calls ixgbe_fc_enable()
to set flowcontrol appropriately.
The other portion of the fix that's core to this is that previously, if the link was
up we were always calling ixgbe_fc_enable() every time through ixgbe_driver_link_check()
which is called once per second based on a ddi periodic function call.
At best you really should only call ixgbe_fc_enable() inside that function
when the link state changes.
The driver does this properly during boot (seen with debug) such that when
the link comes up ixgbe_driver_link_check() enters the "if (link_up)"
conditional and now handles setting FC properly. Reboots with various FC settings
such as 'no' or 'bi' seem to work properly as do changes to FC on the
fly with dladm.
With the call to ixgbe_fc_enable() happening each pass (every second!) through
ixgbe_driver_link_check() it had a negative effect on the X520 network
interfaces ability to communicate. Without an analyzer and more time to dig
it is my suspicion that the constant negociation was interrupting the
cards ability to communicate. The code was doing this every second:
ixgbe_driver_link_check() in ixgbe_main.c ixgbe_fc_enable(hw) in ixgbe_api.c ixgbe_call_func( hw->mac.ops.fc_enable ) macro in ixgbe_type.h calls func filled in by ixgbe_init_ops_generic() in ixgbe_common.c such that it calls: ixgbe_fc_enable_generic() ixgbe_fc_enable_generic() in ixgbe_common.c ixgbe_fc_autoneg() ...and more lower level functions for autoneg such that the chip was negociating with the remote end constantly.
The only other fix in the code is an adjustment of flow control hi-water marks.
Will follow up soon with review request to developer list.
Updated by Kevin Crowe almost 5 years ago
As Dan noted on one of the illumos lists, this fix has been in use inside Nexenta since Fall 2013:
But if anyone wants to pursue that fix they should follow-up on RM's code review feedback here which I ran out of time for: