Project

General

Profile

Actions

Feature #14290

closed

cxgbe: need link speed change when autoneg is disabled

Added by Rahul Lakkireddy 8 months ago. Updated 5 months ago.

Status:
Closed
Priority:
Normal
Category:
driver - device drivers
Start date:
Due date:
% Done:

100%

Estimated time:
Difficulty:
Medium
Tags:
Gerrit CR:

Description

Need ability to change link speeds via dladm for port modules that do not support auto-negotiation or when auto-negotiation is disabled.

Actions #1

Updated by Electric Monk 8 months ago

  • Gerrit CR set to 1849
Actions #2

Updated by Rahul Lakkireddy 8 months ago

Tested changing link speeds using dladm, as shown below.

The Chelsio T6225-SO-CR adapter's port is connected to the PEER using 25G port optics module with 10G fibre optics cable. Auto-negotiation is not available on this port module and link fails to come up. Trying to set 10G speed also fails.

# dladm show-linkprop cxgbe1
LINK         PROPERTY        PERM VALUE          DEFAULT        POSSIBLE
cxgbe1       speed           r-   0              0              -- 
cxgbe1       autopush        rw   --             --             -- 
cxgbe1       zone            rw   --             --             -- 
cxgbe1       duplex          r-   unknown        unknown        half,full 
cxgbe1       state           r-   down           up             up,down 
cxgbe1       adv_autoneg_cap r-   0              0              1,0 
cxgbe1       mtu             rw   1500           1500           46-9600 
cxgbe1       flowctrl        rw   bi             bi             no,tx,rx,bi 
cxgbe1       adv_fec_cap     r-   rs             auto           none,auto,rs,base-r 
cxgbe1       en_fec_cap      rw   auto           auto           none,auto,rs,base-r 
cxgbe1       adv_100gfdx_cap r-   0              0              1,0 
cxgbe1       en_100gfdx_cap  rw   0              0              1,0 
cxgbe1       adv_50gfdx_cap  r-   --             0              1,0 
cxgbe1       en_50gfdx_cap   --   --             0              1,0 
cxgbe1       adv_40gfdx_cap  r-   0              0              1,0 
cxgbe1       en_40gfdx_cap   rw   0              0              1,0 
cxgbe1       adv_25gfdx_cap  r-   1              0              1,0 
cxgbe1       en_25gfdx_cap   rw   1              0              1,0 
cxgbe1       adv_10gfdx_cap  r-   0              0              1,0 
cxgbe1       en_10gfdx_cap   r-   0              0              1,0 
cxgbe1       adv_5000fdx_cap r-   --             0              1,0 
cxgbe1       en_5000fdx_cap  --   --             0              1,0 
cxgbe1       adv_2500fdx_cap r-   --             0              1,0 
cxgbe1       en_2500fdx_cap  --   --             0              1,0 
cxgbe1       adv_1000fdx_cap r-   0              0              1,0 
cxgbe1       en_1000fdx_cap  r-   0              0              1,0 
cxgbe1       adv_1000hdx_cap r-   --             0              1,0 
cxgbe1       en_1000hdx_cap  --   --             0              1,0 
cxgbe1       adv_100fdx_cap  r-   0              0              1,0 
cxgbe1       en_100fdx_cap   r-   0              0              1,0 
cxgbe1       adv_100hdx_cap  r-   --             0              1,0 
cxgbe1       en_100hdx_cap   --   --             0              1,0 
cxgbe1       adv_10fdx_cap   r-   --             0              1,0 
cxgbe1       en_10fdx_cap    --   --             0              1,0 
cxgbe1       adv_10hdx_cap   r-   --             0              1,0 
cxgbe1       en_10hdx_cap    --   --             0              1,0 
cxgbe1       maxbw           rw   --             --             -- 
cxgbe1       cpus            rw   --             --             -- 
cxgbe1       cpus-effective  r-   --             --             -- 
cxgbe1       pool            rw   --             --             -- 
cxgbe1       pool-effective  r-   --             --             -- 
cxgbe1       priority        rw   high           high           low,medium,high 
cxgbe1       tagmode         rw   vlanonly       vlanonly       normal,vlanonly 
cxgbe1       forward         rw   1              1              1,0 
cxgbe1       default_tag     rw   1              1              -- 
cxgbe1       learn_limit     rw   1000           1000           -- 
cxgbe1       learn_decay     rw   200            200            -- 
cxgbe1       stp             rw   1              1              1,0 
cxgbe1       stp_priority    rw   128            128            -- 
cxgbe1       stp_cost        rw   auto           auto           -- 
cxgbe1       stp_edge        rw   1              1              1,0 
LINK         PROPERTY        PERM VALUE          DEFAULT        POSSIBLE
cxgbe1       stp_p2p         rw   auto           auto           true,false,auto 
cxgbe1       stp_mcheck      rw   0              0              1,0 
cxgbe1       protection      rw   --             --             mac-nospoof,
                                                                restricted,
                                                                ip-nospoof,
                                                                dhcp-nospoof 
cxgbe1       allowed-ips     rw   --             --             -- 
cxgbe1       allowed-dhcp-cids rw --             --             -- 
cxgbe1       rxrings         rw   --             --             -- 
cxgbe1       rxrings-effective r- --             --             -- 
cxgbe1       txrings         rw   --             --             sw,hw,<1-3> 
cxgbe1       txrings-effective r- --             --             -- 
cxgbe1       txrings-available r- 3              --             -- 
cxgbe1       rxrings-available r- 0              --             -- 
cxgbe1       rxhwclnt-available r- 0             --             -- 
cxgbe1       txhwclnt-available r- 3             --             -- 
# dladm set-linkprop -t -p en_10gfdx_cap=1,en_25gfdx_cap=0 cxgbe1
dladm: warning: cannot set link property 'en_10gfdx_cap' on 'cxgbe1': operation not supported
dladm: warning: cannot set link property 'en_25gfdx_cap' on 'cxgbe1': operation not supported

With the patch, repeated the same experiment. Able to successfully set 10G speed and link also comes up at 10G speed, as shown below.

# dladm show-linkprop cxgbe1
LINK         PROPERTY        PERM VALUE          DEFAULT        POSSIBLE
cxgbe1       speed           r-   0              0              -- 
cxgbe1       autopush        rw   --             --             -- 
cxgbe1       zone            rw   --             --             -- 
cxgbe1       duplex          r-   unknown        unknown        half,full 
cxgbe1       state           r-   down           up             up,down 
cxgbe1       adv_autoneg_cap r-   0              0              1,0 
cxgbe1       mtu             rw   1500           1500           46-9600 
cxgbe1       flowctrl        rw   bi             bi             no,tx,rx,bi 
cxgbe1       adv_fec_cap     r-   rs             auto           none,auto,rs,base-r 
cxgbe1       en_fec_cap      rw   auto           auto           none,auto,rs,base-r 
cxgbe1       adv_100gfdx_cap r-   0              0              1,0 
cxgbe1       en_100gfdx_cap  r-   0              0              1,0 
cxgbe1       adv_50gfdx_cap  r-   0              0              1,0 
cxgbe1       en_50gfdx_cap   r-   0              0              1,0 
cxgbe1       adv_40gfdx_cap  r-   0              0              1,0 
cxgbe1       en_40gfdx_cap   r-   0              0              1,0 
cxgbe1       adv_25gfdx_cap  r-   1              0              1,0 
cxgbe1       en_25gfdx_cap   rw   1              1              1,0 
cxgbe1       adv_10gfdx_cap  r-   0              0              1,0 
cxgbe1       en_10gfdx_cap   rw   0              1              1,0 
cxgbe1       adv_5000fdx_cap r-   --             0              1,0 
cxgbe1       en_5000fdx_cap  --   --             0              1,0 
cxgbe1       adv_2500fdx_cap r-   --             0              1,0 
cxgbe1       en_2500fdx_cap  --   --             0              1,0 
cxgbe1       adv_1000fdx_cap r-   0              0              1,0 
cxgbe1       en_1000fdx_cap  rw   0              1              1,0 
cxgbe1       adv_1000hdx_cap r-   --             0              1,0 
cxgbe1       en_1000hdx_cap  --   --             0              1,0 
cxgbe1       adv_100fdx_cap  r-   0              0              1,0 
cxgbe1       en_100fdx_cap   r-   0              0              1,0 
cxgbe1       adv_100hdx_cap  r-   --             0              1,0 
cxgbe1       en_100hdx_cap   --   --             0              1,0 
cxgbe1       adv_10fdx_cap   r-   --             0              1,0 
cxgbe1       en_10fdx_cap    --   --             0              1,0 
cxgbe1       adv_10hdx_cap   r-   --             0              1,0 
cxgbe1       en_10hdx_cap    --   --             0              1,0 
cxgbe1       maxbw           rw   --             --             -- 
cxgbe1       cpus            rw   --             --             -- 
cxgbe1       cpus-effective  r-   --             --             -- 
cxgbe1       pool            rw   --             --             -- 
cxgbe1       pool-effective  r-   --             --             -- 
cxgbe1       priority        rw   high           high           low,medium,high 
cxgbe1       tagmode         rw   vlanonly       vlanonly       normal,vlanonly 
cxgbe1       forward         rw   1              1              1,0 
cxgbe1       default_tag     rw   1              1              -- 
cxgbe1       learn_limit     rw   1000           1000           -- 
cxgbe1       learn_decay     rw   200            200            -- 
cxgbe1       stp             rw   1              1              1,0 
cxgbe1       stp_priority    rw   128            128            -- 
cxgbe1       stp_cost        rw   auto           auto           -- 
cxgbe1       stp_edge        rw   1              1              1,0 
LINK         PROPERTY        PERM VALUE          DEFAULT        POSSIBLE
cxgbe1       stp_p2p         rw   auto           auto           true,false,auto 
cxgbe1       stp_mcheck      rw   0              0              1,0 
cxgbe1       protection      rw   --             --             mac-nospoof,
                                                                restricted,
                                                                ip-nospoof,
                                                                dhcp-nospoof 
cxgbe1       allowed-ips     rw   --             --             -- 
cxgbe1       allowed-dhcp-cids rw --             --             -- 
cxgbe1       rxrings         rw   --             --             -- 
cxgbe1       rxrings-effective r- --             --             -- 
cxgbe1       txrings         rw   --             --             sw,hw,<1-3> 
cxgbe1       txrings-effective r- --             --             -- 
cxgbe1       txrings-available r- 3              --             -- 
cxgbe1       rxrings-available r- 0              --             -- 
cxgbe1       rxhwclnt-available r- 0             --             -- 
cxgbe1       txhwclnt-available r- 3             --             -- 
# dladm set-linkprop -t -p en_10gfdx_cap=1,en_25gfdx_cap=0 cxgbe1
# dladm show-linkprop cxgbe1
LINK         PROPERTY        PERM VALUE          DEFAULT        POSSIBLE
cxgbe1       speed           r-   10000          10000          -- 
cxgbe1       autopush        rw   --             --             -- 
cxgbe1       zone            rw   --             --             -- 
cxgbe1       duplex          r-   full           full           half,full 
cxgbe1       state           r-   up             up             up,down 
cxgbe1       adv_autoneg_cap r-   0              0              1,0 
cxgbe1       mtu             rw   1500           1500           46-9600 
cxgbe1       flowctrl        rw   bi             bi             no,tx,rx,bi 
cxgbe1       adv_fec_cap     r-   none           auto           none,auto,rs,base-r 
cxgbe1       en_fec_cap      rw   none           auto           none,auto,rs,base-r 
cxgbe1       adv_100gfdx_cap r-   0              0              1,0 
cxgbe1       en_100gfdx_cap  r-   0              0              1,0 
cxgbe1       adv_50gfdx_cap  r-   0              0              1,0 
cxgbe1       en_50gfdx_cap   r-   0              0              1,0 
cxgbe1       adv_40gfdx_cap  r-   0              0              1,0 
cxgbe1       en_40gfdx_cap   r-   0              0              1,0 
cxgbe1       adv_25gfdx_cap  r-   0              0              1,0 
cxgbe1       en_25gfdx_cap   rw   0              1              1,0 
cxgbe1       adv_10gfdx_cap  r-   1              0              1,0 
cxgbe1       en_10gfdx_cap   rw   1              1              1,0 
cxgbe1       adv_5000fdx_cap r-   --             0              1,0 
cxgbe1       en_5000fdx_cap  --   --             0              1,0 
cxgbe1       adv_2500fdx_cap r-   --             0              1,0 
cxgbe1       en_2500fdx_cap  --   --             0              1,0 
cxgbe1       adv_1000fdx_cap r-   0              0              1,0 
cxgbe1       en_1000fdx_cap  rw   1              1              1,0 
cxgbe1       adv_1000hdx_cap r-   --             0              1,0 
cxgbe1       en_1000hdx_cap  --   --             0              1,0 
cxgbe1       adv_100fdx_cap  r-   0              0              1,0 
cxgbe1       en_100fdx_cap   r-   0              0              1,0 
cxgbe1       adv_100hdx_cap  r-   --             0              1,0 
cxgbe1       en_100hdx_cap   --   --             0              1,0 
cxgbe1       adv_10fdx_cap   r-   --             0              1,0 
cxgbe1       en_10fdx_cap    --   --             0              1,0 
cxgbe1       adv_10hdx_cap   r-   --             0              1,0 
cxgbe1       en_10hdx_cap    --   --             0              1,0 
cxgbe1       maxbw           rw   --             --             -- 
cxgbe1       cpus            rw   --             --             -- 
cxgbe1       cpus-effective  r-   0-3            --             -- 
cxgbe1       pool            rw   --             --             -- 
cxgbe1       pool-effective  r-   --             --             -- 
cxgbe1       priority        rw   high           high           low,medium,high 
cxgbe1       tagmode         rw   vlanonly       vlanonly       normal,vlanonly 
cxgbe1       forward         rw   1              1              1,0 
cxgbe1       default_tag     rw   1              1              -- 
cxgbe1       learn_limit     rw   1000           1000           -- 
cxgbe1       learn_decay     rw   200            200            -- 
cxgbe1       stp             rw   1              1              1,0 
cxgbe1       stp_priority    rw   128            128            -- 
cxgbe1       stp_cost        rw   auto           auto           -- 
cxgbe1       stp_edge        rw   1              1              1,0 
LINK         PROPERTY        PERM VALUE          DEFAULT        POSSIBLE
cxgbe1       stp_p2p         rw   auto           auto           true,false,auto 
cxgbe1       stp_mcheck      rw   0              0              1,0 
cxgbe1       protection      rw   --             --             mac-nospoof,
                                                                restricted,
                                                                ip-nospoof,
                                                                dhcp-nospoof 
cxgbe1       allowed-ips     rw   --             --             -- 
cxgbe1       allowed-dhcp-cids rw --             --             -- 
cxgbe1       rxrings         rw   --             --             -- 
cxgbe1       rxrings-effective r- 3              --             -- 
cxgbe1       txrings         rw   --             --             sw,hw,<1-3> 
cxgbe1       txrings-effective r- 4              --             -- 
cxgbe1       txrings-available r- 3              --             -- 
cxgbe1       rxrings-available r- 0              --             -- 
cxgbe1       rxhwclnt-available r- 0             --             -- 
cxgbe1       txhwclnt-available r- 3             --             --
Actions #3

Updated by Rahul Lakkireddy 8 months ago

Following changes are added in the patch:

1) Rework and group link related common code to ease translating link params from t4nex mac registered callbacks to internal firmware's format.

2) Rework t4nex link related mac callbacks to take advantage of rework done in common code. This rework enables changing link speed parameters for port modules that do not support autonegotiation or when autonegotiation is disabled using common code.

3) Adds support to t4nex to allow changing link speeds to 25G/40G/50G/100G via dladm.

4) Enable synchronized access to mailbox. The code is already present, but had been placed under #ifdef T4_OS_NEEDS_MBOX_LOCKING, which is not defined and hence the related functions were no-op. So, they are moved out of T4_OS_NEEDS_MBOX_LOCKING and the related functions have been implemented.

The maibox access needs to be protected from simultaneous access. This can happen when a link update comes from firmware via interrupt during first link up or when port module is changed/replugged-in. When this link update comes, an attempt is made to resend the user configured link params on the new plugged-in port module via mailbox. This overwrites the current pending operation in the mailbox (which is waiting for the reply), and the mailbox gets corrupted.

5) Fixes a use-after-free issue, when multiple Chelsio adapters are detached. Ensure to remove the adapter from t4_adapter_list SLIST in the same order they are detached, instead of always removing from SLIST head.

Thanks,
Rahul

Actions #4

Updated by Electric Monk 5 months ago

  • Status changed from New to Closed
  • % Done changed from 0 to 100

git commit 6feac2e3ccf4b4d8651a254d516ae86b1b1b3e4f

commit  6feac2e3ccf4b4d8651a254d516ae86b1b1b3e4f
Author: Rahul Lakkireddy <rahul.lakkireddy@chelsio.com>
Date:   2022-03-05T00:39:23.000Z

    14290 cxgbe: need link speed change when autoneg is disabled
    Reviewed by: Jason King <jason.brian.king+illumos@gmail.com>
    Reviewed by: Ryan Goodfellow <ryan.goodfellow@oxide.computer>
    Approved by: Richard Lowe <richlowe@richlowe.net>

Actions

Also available in: Atom PDF