illumos gate

I have a superMicro X11SPH-nCTF(1) and the onboard nics don't work.

They are in an unknown state when coming from a cold boot:

[root@boron ~]# dladm show-phys
LINK         MEDIA               STATE      SPEED  DUPLEX    DEVICE
i40e0        Ethernet             unknown    0      unknown   i40e0
i40e1        Ethernet             unknown    0      unknown   i40e1
ixgbe0       Ethernet             up         10000  full      ixgbe0
igb0         Ethernet             up         1000   full      igb0

The switch seems the as negotiated to 1Gbit FDX (which is correct), when resetting the box once it is been up. Both come up as 1000 full as expected.

Looking at the source(3) and X557 PHYs spec sheet (2) this makes sense, the device from cold boot will start in extended SGMII auto-negotiation, when looking a the code this ends up in the unknown state. After a reset while the links are up they are in the 1000BASE-X interface on (SGMII without auto-negotiation) state, so we do end up in the correct driver state!

From the X557 PHY linked below:

1.3 SerDes
The X557 is designed to be used in conjunction with a triple-speed MAC chip:
• KR/XFI for 10 GbE
• 1000BASE-X for 1 GbE (SerDes logical lane 0)
• SGMII for 100 Mb/s (SerDes logical lane 0)
Note: The X557-AT2 10 GbE PHY internally has four SerDes per PHY, but only physical lane 2 is
connected. Consequently, the register map contains information for lanes 0, 1 and 3, but
these cannot be used in the X557-AT2. The X557 is capable of being configured so that the
MAC interface can start in one of the following modes:
1. Start with the 10 GbE interface on KR/XFI.
2. Start with the 1000BASE-X interface on (SGMII without auto-negotiation).
3. Start with extended SGMII auto-negotiation enabled.
4. Start with all interfaces off.
Once a connection to a link partner has been established, the interface switches to the correct MAC
interface speed.

Although a X722 with SPF+ PHY doesn't seem to support 100 FDX/SGMII... with a X557 PHY attached it does seem to end up in this mode.
Forcing the switch to 100M FDX makes the link also come up, but after a reset the box remains in an unknown state... I'm not sure what I need to change to make it work from a cold boot.

1) [[https://www.supermicro.com/en/products/motherboard/X11SPH-nCTF]]
2) [[https://cdrdv2.intel.com/v1/dl/getcontent/334279]]
3) [[http://src.illumos.org/source/xref/illumos-gate/usr/src/uts/common/io/i40e/core/i40e_common.c#1243]]