illumos gate

It looks like this bug is in the VLAN layer, not IPMP:

# dladm
LINK        CLASS     MTU    STATE    BRIDGE     OVER
igb0        phys      1500   up       --         --
igb1        phys      1500   unknown  --         --
igb6        phys      1500   unknown  --         --
igb4        phys      1500   unknown  --         --
igb2        phys      1500   unknown  --         --
igb3        phys      1500   unknown  --         --
ixgbe0      phys      1500   down     --         --
igb7        phys      1500   unknown  --         --
igb5        phys      1500   unknown  --         --
ixgbe1      phys      1500   up       --         --
vlan326v0   vlan      1500   down     --         ixgbe0
vlan326v1   vlan      1500   up       --         ixgbe1
vlan2308v0  vlan      1500   down     --         ixgbe0
vlan2308v1  vlan      1500   up       --         ixgbe1
vlan2342vnic0 vnic    1500   up       --         ixgbe0
vlan2342vnic1 vnic    1500   up       --         ixgbe1
vlan2307v1  vlan      1500   up       --         ixgbe1
vlan2307v0  vlan      1500   down     --         ixgbe0
vlan2342v1  vlan      1500   up       --         ixgbe1
vlan2342v0  vlan      1500   up       --         ixgbe0

This is weird.

There are two ways to create a "VLAN Nic". One is via `dladm create-vlan -v vid -l physN linkN`, the other is via `dladm create-vnic -v vid -l physN linkN`. The one that isn't failing properly for you was created by `create-vnic`. So the question is: What's different about that such that it doesn't propagate the failure?

IF you have your machine in this state still, a `savecore -L` should have the state one would need to dive into the internal structures and see what's going on, and it won't force a reboot. Also, for people reproducing this bug (including me some time this week), the experiment worth trying is:

1.) Have two physical same-link NICs on your test machine.
2.) Create one vlan over each using `dladm create-vlan...`
3.) Create one VNIC with a vlan id over each using `dladm create-vnic`
4.) Have an off-link peer where you can send packets over each vlan.
5.) Start moving bits.
6.) Unplug (virtually or otherwise) one of the nics.
7.) See if you end up in the state Chip describes.

ALSO NOTE: This is on OmniOSce r151030ap. It may be necessary to reproduce this on that release or older alongside a current one, just in case it's a bug that's been fixed (e.g. #12355 maybe?).

I configured a VMware VM using e1000g nics. I cannot, on 030ap, reproduce this. I'm not sure if it's e1000g vs. ixgbe, or dumb luck, or some other problem. I did mention 12355 earlier, but DTrace suggests THAT isn't tickled during NIC removal/restart.

# dladm show-link ; ipmpstat -g
LINK        CLASS     MTU    STATE    BRIDGE     OVER
e1000g0     phys      1500   up       --         --
e1000g1     phys      1500   up       --         --
vh84l0      vlan      1500   up       --         e1000g0
vh84l1      vlan      1500   up       --         e1000g1
vh86l0      vnic      1500   up       --         e1000g0
vh86l1      vnic      1500   up       --         e1000g1
GROUP       GROUPNAME   STATE     FDT       INTERFACES
sammy0      sammy0      ok        --        vh86l1 vh86l0
dave0       dave0       ok        --        vh84l1 vh84l0
<XXX REMOVE e1000g0, delay because this is over ssh over same link>
# sleep 10 ; dladm show-link ; ipmpstat -g
LINK        CLASS     MTU    STATE    BRIDGE     OVER
e1000g0     phys      1500   down     --         --
e1000g1     phys      1500   up       --         --
vh84l0      vlan      1500   down     --         e1000g0
vh84l1      vlan      1500   up       --         e1000g1
vh86l0      vnic      1500   down     --         e1000g0
vh86l1      vnic      1500   up       --         e1000g1
GROUP       GROUPNAME   STATE     FDT       INTERFACES
sammy0      sammy0      degraded  --        vh86l1 [vh86l0]
dave0       dave0       degraded  --        vh84l1 [vh84l0]
<XXX RE-ADD e1000g0>
# dladm show-link ; ipmpstat -g
LINK        CLASS     MTU    STATE    BRIDGE     OVER
e1000g0     phys      1500   up       --         --
e1000g1     phys      1500   up       --         --
vh84l0      vlan      1500   up       --         e1000g0
vh84l1      vlan      1500   up       --         e1000g1
vh86l0      vnic      1500   up       --         e1000g0
vh86l1      vnic      1500   up       --         e1000g1
GROUP       GROUPNAME   STATE     FDT       INTERFACES
sammy0      sammy0      ok        --        vh86l1 vh86l0
dave0       dave0       ok        --        vh84l1 vh84l0
<XXX REMOVE e1000g1>
# dladm show-link ; ipmpstat -g
LINK        CLASS     MTU    STATE    BRIDGE     OVER
e1000g0     phys      1500   up       --         --
e1000g1     phys      1500   down     --         --
vh84l0      vlan      1500   up       --         e1000g0
vh84l1      vlan      1500   down     --         e1000g1
vh86l0      vnic      1500   up       --         e1000g0
vh86l1      vnic      1500   down     --         e1000g1
GROUP       GROUPNAME   STATE     FDT       INTERFACES
sammy0      sammy0      degraded  --        vh86l0 [vh86l1]
dave0       dave0       degraded  --        vh84l0 [vh84l1]
<XXX RE-ADD e1000g1>
# dladm show-link ; ipmpstat -g
LINK        CLASS     MTU    STATE    BRIDGE     OVER
e1000g0     phys      1500   up       --         --
e1000g1     phys      1500   up       --         --
vh84l0      vlan      1500   up       --         e1000g0
vh84l1      vlan      1500   up       --         e1000g1
vh86l0      vnic      1500   up       --         e1000g0
vh86l1      vnic      1500   up       --         e1000g1
GROUP       GROUPNAME   STATE     FDT       INTERFACES
sammy0      sammy0      ok        --        vh86l1 vh86l0
dave0       dave0       ok        --        vh84l1 vh84l0
# 

I figured out how to reproduce this. The presence of vnics on the same interfaces will cause the creation of VLANs in an up state on down interfaces.

I'm collecting a savecore now and will update this ticket with a place to download it.

The savecore file can be downloaded at:

https://repo.nrg.wustl.edu/pub/zfs/vmdump_ipmp_down.0

Found the problem:

http://src.illumos.org/source/xref/illumos-gate/usr/src/uts/common/io/mac/mac_client.c#547

And I quote:

547/*
548 * Return the link state of the specified client. If here are more
549 * than one clients of the underying mac_impl_t, the link state
550 * will always be UP regardless of the link state of the underlying
551 * mac_impl_t. This is needed to allow the MAC clients to continue
552 * to communicate with each other even when the physical link of
553 * their mac_impl_t is down.
554 */

and

563    /*
564     * Returns LINK_STATE_UP if there are other MAC clients defined on
565     * mac_impl_t which share same VLAN ID as that of mcip. Note that
566     * if 'mcip' has more than one VID's then we match ANY one of the
567     * VID's with other MAC client's VID's and return LINK_STATE_UP.
568     */

For a unshared-VLAN-ID vnic, the call stack to alter the link state is as follows:

  0   | vnic_notify_cb:entry                  
  0    -> mac_client_stat_get                 
  0      -> mac_client_link_state             
  0        -> mac_stat_get                    
  0        <- mac_stat_get                    Returns 0x0
  0      <- mac_client_link_state             Returns 0x0
  0    <- mac_client_stat_get                 Returns 0x0 <==== KEBE SAYS 0 is DOWN

For a shared one, it's different:

  0   | vnic_notify_cb:entry                  
  0    -> mac_client_stat_get                 
  0      -> mac_client_link_state             
  0      <- mac_client_link_state             Returns 0x1
  0    <- mac_client_stat_get                 Returns 0x1 <==== KEBE SAYS 1 is UP
  0    -> mac_link_update                     

For an IPMP group, this is bad, per the original report.

For every other use of VNICs or VLANs, including and especially communicating with each other (e.g. two zones with two vnics on the same VLAN ID shouldn't get cut off from each other because the underlying physical NIC got unplugged), this is actually good, as it does not prevent inter-zone/intra-machine communication.

I'm lowering the priority of this bug for now, as the filer indicated the duplicate VLAN ID was a mistake/typo. I won't close this yet, but I'm leaning toward a "works as designed/not a bug" designation. I'd like the filer, and anyone else following this bug, to comment.

Hi!
I have an issue on my host with 2 zones/vnics that might be related.

If the physical port is down for some reason,
and reconnects after a while,
i think the vnics will not be available again via the physical link.

i stick to vlan id 0 always as i don't have/need vlan's.

i have to verify my assumption when i get physical access to the machine somewhen..

Actually using the same VLAN was not a mistake. In this case, I had previously stopped the LX zone, but didn't remove the vnic.

Any time I run LX zones, I have to use vnics. These will always share a vlan with other existing NFS services which are shared via IPs attached to IPMP over vlans, not vnics. Both network stacks need to function simultaneously.

Stephan Althaus wrote:

Hi!
I have an issue on my host with 2 zones/vnics that might be related.

If the physical port is down for some reason,
and reconnects after a while,
i think the vnics will not be available again via the physical link.

i stick to vlan id 0 always as i don't have/need vlan's.

i have to verify my assumption when i get physical access to the machine somewhen..

Outputs of:

dladm show-link
dladm show-ether
dladm show-vnic

will be helpful.

Chip Schweiss wrote:

Any time I run LX zones, I have to use vnics. These will always share a vlan with other existing NFS services which are shared via IPs attached to IPMP over vlans, not vnics. Both network stacks need to function simultaneously.

Without serious rearchitecture, the tradeoff stands as follows:

1.) (today's state) Multiple-same-VLAN-vnics won't go down if the lower-link does, so inter-vnic/intra-machine communication can continue.

2.) (an easy candidate for a tunable) Multiple-same-VLAN-vnics WILL go down if the lower-link does, breaking inter-vnic/intra-machine communication, BUT enabling failover/detection mechanisms (like IPMP) to do the right thing.

So if you want to run any type of zone attached to a vlan, you don't get to have any sort of bonded network on the same vlan.

This would break LCAP just the same.

My vote is for #2, which should be the default behavior when network bonding is in place.