Feature #14290
closed
cxgbe: need link speed change when autoneg is disabled
100%
Description
Need ability to change link speeds via dladm for port modules that do not support auto-negotiation or when auto-negotiation is disabled.
Updated by Rahul Lakkireddy 8 months ago
Tested changing link speeds using dladm, as shown below.
The Chelsio T6225-SO-CR adapter's port is connected to the PEER using 25G port optics module with 10G fibre optics cable. Auto-negotiation is not available on this port module and link fails to come up. Trying to set 10G speed also fails.
# dladm show-linkprop cxgbe1
LINK PROPERTY PERM VALUE DEFAULT POSSIBLE
cxgbe1 speed r- 0 0 --
cxgbe1 autopush rw -- -- --
cxgbe1 zone rw -- -- --
cxgbe1 duplex r- unknown unknown half,full
cxgbe1 state r- down up up,down
cxgbe1 adv_autoneg_cap r- 0 0 1,0
cxgbe1 mtu rw 1500 1500 46-9600
cxgbe1 flowctrl rw bi bi no,tx,rx,bi
cxgbe1 adv_fec_cap r- rs auto none,auto,rs,base-r
cxgbe1 en_fec_cap rw auto auto none,auto,rs,base-r
cxgbe1 adv_100gfdx_cap r- 0 0 1,0
cxgbe1 en_100gfdx_cap rw 0 0 1,0
cxgbe1 adv_50gfdx_cap r- -- 0 1,0
cxgbe1 en_50gfdx_cap -- -- 0 1,0
cxgbe1 adv_40gfdx_cap r- 0 0 1,0
cxgbe1 en_40gfdx_cap rw 0 0 1,0
cxgbe1 adv_25gfdx_cap r- 1 0 1,0
cxgbe1 en_25gfdx_cap rw 1 0 1,0
cxgbe1 adv_10gfdx_cap r- 0 0 1,0
cxgbe1 en_10gfdx_cap r- 0 0 1,0
cxgbe1 adv_5000fdx_cap r- -- 0 1,0
cxgbe1 en_5000fdx_cap -- -- 0 1,0
cxgbe1 adv_2500fdx_cap r- -- 0 1,0
cxgbe1 en_2500fdx_cap -- -- 0 1,0
cxgbe1 adv_1000fdx_cap r- 0 0 1,0
cxgbe1 en_1000fdx_cap r- 0 0 1,0
cxgbe1 adv_1000hdx_cap r- -- 0 1,0
cxgbe1 en_1000hdx_cap -- -- 0 1,0
cxgbe1 adv_100fdx_cap r- 0 0 1,0
cxgbe1 en_100fdx_cap r- 0 0 1,0
cxgbe1 adv_100hdx_cap r- -- 0 1,0
cxgbe1 en_100hdx_cap -- -- 0 1,0
cxgbe1 adv_10fdx_cap r- -- 0 1,0
cxgbe1 en_10fdx_cap -- -- 0 1,0
cxgbe1 adv_10hdx_cap r- -- 0 1,0
cxgbe1 en_10hdx_cap -- -- 0 1,0
cxgbe1 maxbw rw -- -- --
cxgbe1 cpus rw -- -- --
cxgbe1 cpus-effective r- -- -- --
cxgbe1 pool rw -- -- --
cxgbe1 pool-effective r- -- -- --
cxgbe1 priority rw high high low,medium,high
cxgbe1 tagmode rw vlanonly vlanonly normal,vlanonly
cxgbe1 forward rw 1 1 1,0
cxgbe1 default_tag rw 1 1 --
cxgbe1 learn_limit rw 1000 1000 --
cxgbe1 learn_decay rw 200 200 --
cxgbe1 stp rw 1 1 1,0
cxgbe1 stp_priority rw 128 128 --
cxgbe1 stp_cost rw auto auto --
cxgbe1 stp_edge rw 1 1 1,0
LINK PROPERTY PERM VALUE DEFAULT POSSIBLE
cxgbe1 stp_p2p rw auto auto true,false,auto
cxgbe1 stp_mcheck rw 0 0 1,0
cxgbe1 protection rw -- -- mac-nospoof,
restricted,
ip-nospoof,
dhcp-nospoof
cxgbe1 allowed-ips rw -- -- --
cxgbe1 allowed-dhcp-cids rw -- -- --
cxgbe1 rxrings rw -- -- --
cxgbe1 rxrings-effective r- -- -- --
cxgbe1 txrings rw -- -- sw,hw,<1-3>
cxgbe1 txrings-effective r- -- -- --
cxgbe1 txrings-available r- 3 -- --
cxgbe1 rxrings-available r- 0 -- --
cxgbe1 rxhwclnt-available r- 0 -- --
cxgbe1 txhwclnt-available r- 3 -- --
# dladm set-linkprop -t -p en_10gfdx_cap=1,en_25gfdx_cap=0 cxgbe1 dladm: warning: cannot set link property 'en_10gfdx_cap' on 'cxgbe1': operation not supported dladm: warning: cannot set link property 'en_25gfdx_cap' on 'cxgbe1': operation not supported
With the patch, repeated the same experiment. Able to successfully set 10G speed and link also comes up at 10G speed, as shown below.
# dladm show-linkprop cxgbe1
LINK PROPERTY PERM VALUE DEFAULT POSSIBLE
cxgbe1 speed r- 0 0 --
cxgbe1 autopush rw -- -- --
cxgbe1 zone rw -- -- --
cxgbe1 duplex r- unknown unknown half,full
cxgbe1 state r- down up up,down
cxgbe1 adv_autoneg_cap r- 0 0 1,0
cxgbe1 mtu rw 1500 1500 46-9600
cxgbe1 flowctrl rw bi bi no,tx,rx,bi
cxgbe1 adv_fec_cap r- rs auto none,auto,rs,base-r
cxgbe1 en_fec_cap rw auto auto none,auto,rs,base-r
cxgbe1 adv_100gfdx_cap r- 0 0 1,0
cxgbe1 en_100gfdx_cap r- 0 0 1,0
cxgbe1 adv_50gfdx_cap r- 0 0 1,0
cxgbe1 en_50gfdx_cap r- 0 0 1,0
cxgbe1 adv_40gfdx_cap r- 0 0 1,0
cxgbe1 en_40gfdx_cap r- 0 0 1,0
cxgbe1 adv_25gfdx_cap r- 1 0 1,0
cxgbe1 en_25gfdx_cap rw 1 1 1,0
cxgbe1 adv_10gfdx_cap r- 0 0 1,0
cxgbe1 en_10gfdx_cap rw 0 1 1,0
cxgbe1 adv_5000fdx_cap r- -- 0 1,0
cxgbe1 en_5000fdx_cap -- -- 0 1,0
cxgbe1 adv_2500fdx_cap r- -- 0 1,0
cxgbe1 en_2500fdx_cap -- -- 0 1,0
cxgbe1 adv_1000fdx_cap r- 0 0 1,0
cxgbe1 en_1000fdx_cap rw 0 1 1,0
cxgbe1 adv_1000hdx_cap r- -- 0 1,0
cxgbe1 en_1000hdx_cap -- -- 0 1,0
cxgbe1 adv_100fdx_cap r- 0 0 1,0
cxgbe1 en_100fdx_cap r- 0 0 1,0
cxgbe1 adv_100hdx_cap r- -- 0 1,0
cxgbe1 en_100hdx_cap -- -- 0 1,0
cxgbe1 adv_10fdx_cap r- -- 0 1,0
cxgbe1 en_10fdx_cap -- -- 0 1,0
cxgbe1 adv_10hdx_cap r- -- 0 1,0
cxgbe1 en_10hdx_cap -- -- 0 1,0
cxgbe1 maxbw rw -- -- --
cxgbe1 cpus rw -- -- --
cxgbe1 cpus-effective r- -- -- --
cxgbe1 pool rw -- -- --
cxgbe1 pool-effective r- -- -- --
cxgbe1 priority rw high high low,medium,high
cxgbe1 tagmode rw vlanonly vlanonly normal,vlanonly
cxgbe1 forward rw 1 1 1,0
cxgbe1 default_tag rw 1 1 --
cxgbe1 learn_limit rw 1000 1000 --
cxgbe1 learn_decay rw 200 200 --
cxgbe1 stp rw 1 1 1,0
cxgbe1 stp_priority rw 128 128 --
cxgbe1 stp_cost rw auto auto --
cxgbe1 stp_edge rw 1 1 1,0
LINK PROPERTY PERM VALUE DEFAULT POSSIBLE
cxgbe1 stp_p2p rw auto auto true,false,auto
cxgbe1 stp_mcheck rw 0 0 1,0
cxgbe1 protection rw -- -- mac-nospoof,
restricted,
ip-nospoof,
dhcp-nospoof
cxgbe1 allowed-ips rw -- -- --
cxgbe1 allowed-dhcp-cids rw -- -- --
cxgbe1 rxrings rw -- -- --
cxgbe1 rxrings-effective r- -- -- --
cxgbe1 txrings rw -- -- sw,hw,<1-3>
cxgbe1 txrings-effective r- -- -- --
cxgbe1 txrings-available r- 3 -- --
cxgbe1 rxrings-available r- 0 -- --
cxgbe1 rxhwclnt-available r- 0 -- --
cxgbe1 txhwclnt-available r- 3 -- --
# dladm set-linkprop -t -p en_10gfdx_cap=1,en_25gfdx_cap=0 cxgbe1
# dladm show-linkprop cxgbe1
LINK PROPERTY PERM VALUE DEFAULT POSSIBLE
cxgbe1 speed r- 10000 10000 --
cxgbe1 autopush rw -- -- --
cxgbe1 zone rw -- -- --
cxgbe1 duplex r- full full half,full
cxgbe1 state r- up up up,down
cxgbe1 adv_autoneg_cap r- 0 0 1,0
cxgbe1 mtu rw 1500 1500 46-9600
cxgbe1 flowctrl rw bi bi no,tx,rx,bi
cxgbe1 adv_fec_cap r- none auto none,auto,rs,base-r
cxgbe1 en_fec_cap rw none auto none,auto,rs,base-r
cxgbe1 adv_100gfdx_cap r- 0 0 1,0
cxgbe1 en_100gfdx_cap r- 0 0 1,0
cxgbe1 adv_50gfdx_cap r- 0 0 1,0
cxgbe1 en_50gfdx_cap r- 0 0 1,0
cxgbe1 adv_40gfdx_cap r- 0 0 1,0
cxgbe1 en_40gfdx_cap r- 0 0 1,0
cxgbe1 adv_25gfdx_cap r- 0 0 1,0
cxgbe1 en_25gfdx_cap rw 0 1 1,0
cxgbe1 adv_10gfdx_cap r- 1 0 1,0
cxgbe1 en_10gfdx_cap rw 1 1 1,0
cxgbe1 adv_5000fdx_cap r- -- 0 1,0
cxgbe1 en_5000fdx_cap -- -- 0 1,0
cxgbe1 adv_2500fdx_cap r- -- 0 1,0
cxgbe1 en_2500fdx_cap -- -- 0 1,0
cxgbe1 adv_1000fdx_cap r- 0 0 1,0
cxgbe1 en_1000fdx_cap rw 1 1 1,0
cxgbe1 adv_1000hdx_cap r- -- 0 1,0
cxgbe1 en_1000hdx_cap -- -- 0 1,0
cxgbe1 adv_100fdx_cap r- 0 0 1,0
cxgbe1 en_100fdx_cap r- 0 0 1,0
cxgbe1 adv_100hdx_cap r- -- 0 1,0
cxgbe1 en_100hdx_cap -- -- 0 1,0
cxgbe1 adv_10fdx_cap r- -- 0 1,0
cxgbe1 en_10fdx_cap -- -- 0 1,0
cxgbe1 adv_10hdx_cap r- -- 0 1,0
cxgbe1 en_10hdx_cap -- -- 0 1,0
cxgbe1 maxbw rw -- -- --
cxgbe1 cpus rw -- -- --
cxgbe1 cpus-effective r- 0-3 -- --
cxgbe1 pool rw -- -- --
cxgbe1 pool-effective r- -- -- --
cxgbe1 priority rw high high low,medium,high
cxgbe1 tagmode rw vlanonly vlanonly normal,vlanonly
cxgbe1 forward rw 1 1 1,0
cxgbe1 default_tag rw 1 1 --
cxgbe1 learn_limit rw 1000 1000 --
cxgbe1 learn_decay rw 200 200 --
cxgbe1 stp rw 1 1 1,0
cxgbe1 stp_priority rw 128 128 --
cxgbe1 stp_cost rw auto auto --
cxgbe1 stp_edge rw 1 1 1,0
LINK PROPERTY PERM VALUE DEFAULT POSSIBLE
cxgbe1 stp_p2p rw auto auto true,false,auto
cxgbe1 stp_mcheck rw 0 0 1,0
cxgbe1 protection rw -- -- mac-nospoof,
restricted,
ip-nospoof,
dhcp-nospoof
cxgbe1 allowed-ips rw -- -- --
cxgbe1 allowed-dhcp-cids rw -- -- --
cxgbe1 rxrings rw -- -- --
cxgbe1 rxrings-effective r- 3 -- --
cxgbe1 txrings rw -- -- sw,hw,<1-3>
cxgbe1 txrings-effective r- 4 -- --
cxgbe1 txrings-available r- 3 -- --
cxgbe1 rxrings-available r- 0 -- --
cxgbe1 rxhwclnt-available r- 0 -- --
cxgbe1 txhwclnt-available r- 3 -- --
Updated by Rahul Lakkireddy 8 months ago
Following changes are added in the patch:
1) Rework and group link related common code to ease translating link params from t4nex mac registered callbacks to internal firmware's format.
2) Rework t4nex link related mac callbacks to take advantage of rework done in common code. This rework enables changing link speed parameters for port modules that do not support autonegotiation or when autonegotiation is disabled using common code.
3) Adds support to t4nex to allow changing link speeds to 25G/40G/50G/100G via dladm.
4) Enable synchronized access to mailbox. The code is already present, but had been placed under #ifdef T4_OS_NEEDS_MBOX_LOCKING, which is not defined and hence the related functions were no-op. So, they are moved out of T4_OS_NEEDS_MBOX_LOCKING and the related functions have been implemented.
The maibox access needs to be protected from simultaneous access. This can happen when a link update comes from firmware via interrupt during first link up or when port module is changed/replugged-in. When this link update comes, an attempt is made to resend the user configured link params on the new plugged-in port module via mailbox. This overwrites the current pending operation in the mailbox (which is waiting for the reply), and the mailbox gets corrupted.
5) Fixes a use-after-free issue, when multiple Chelsio adapters are detached. Ensure to remove the adapter from t4_adapter_list SLIST in the same order they are detached, instead of always removing from SLIST head.
Thanks,
Rahul
Updated by Electric Monk 5 months ago
- Status changed from New to Closed
- % Done changed from 0 to 100
git commit 6feac2e3ccf4b4d8651a254d516ae86b1b1b3e4f
commit 6feac2e3ccf4b4d8651a254d516ae86b1b1b3e4f
Author: Rahul Lakkireddy <rahul.lakkireddy@chelsio.com>
Date: 2022-03-05T00:39:23.000Z
14290 cxgbe: need link speed change when autoneg is disabled
Reviewed by: Jason King <jason.brian.king+illumos@gmail.com>
Reviewed by: Ryan Goodfellow <ryan.goodfellow@oxide.computer>
Approved by: Richard Lowe <richlowe@richlowe.net>