Project

General

Profile

Bug #2869

duplicate packets with vnics over aggrs

Added by Theo Schlossnagle over 8 years ago. Updated 9 months ago.

Status:
Closed
Priority:
Normal
Assignee:
-
Category:
kernel
Start date:
2012-06-12
Due date:
% Done:

100%

Estimated time:
Difficulty:
Medium
Tags:
needs-triage
Gerrit CR:

Description

On a current system with two e1000g NICs in an LACP aggregate, on which vnics are placed, we see duplicate packets. (simple ping will show DUP responses).

Repeat reports in igb are here: http://blogs.everycity.co.uk/alasdair/2010/10/weird-opensolariscrossbow-issue-with-aggregations-and-vlansvnics/
and bge here: http://mail.opensolaris.org/pipermail/crossbow-discuss/2009-June/001923.html

A user claims it was fixed in Solaris 11 GA (not that it helps).

#1

Updated by Rich Lowe over 8 years ago

  • Subject changed from duplicate packets to duplicate packets with vnics over aggrs
#2

Updated by Theo Schlossnagle over 8 years ago

  • % Done changed from 0 to 20

The issue appears to fall along the lines of hardware support.

If the underlying ethernet devices support hardware MAC matching, everything works. When a device is flipped into promiscuous mode, packets are delivered via mac_promisc_dispatch and then not delivered by the hardware matching code path.

On the other hand, in software mode, we fallback to pushing packets through flows. However, there is no way for the software flow code to know whether it should be pushing the packets in (as that in an impl detail).

On our systems with e1000g devices, everything works fine as hardware MAC matching is supported and the aggr never flips the underlying interfaces into promiscuous mode. On our nge-based machines, aggr will set the underlying interfaces into promiscuous mode an a packet wil be delivered twice in mac_rx_common via mac_promisc_dispatch and then again via (flent->fe_cb_fn)(flent->fe_cb_arg1, flent->fe_cb_arg2, mp_chain, B_FALSE);

I propose making the aggr code aware of this and avoiding it at that layer. Only the aggr layer knows whether it has put the interfaces into promiscuous mode for "ordinary" purposes.

#3

Updated by Theo Schlossnagle over 8 years ago

For reference, here is a proposed fix: http://cr.illumos.org/~webrev/postwait/2869/

This fix is not optimal as the mac layer is still producing two chains, one of which we are freeing.

It is possible that mac_rx_set and mac_rx_clear are malfunctioning and the aggr is not correctly registering itself/deregistering itself, but I think the interplay with VNICs atop them makes that unclear.

I've been unable to diagnose malfunctioning in the MAC layer and have not been successful removing the production of the duplicate chain of packets. (the above fix simply suppresses them before they are delivered through the aggr)

#4

Updated by Dan McDonald about 7 years ago

Previously, it was stated that this bug manifests with devices in the aggr that don't support MAC matching in hardware.

Assuming so, is there a whitelist of HW where this problem does not occur? I just tried to reproduce this with Intel I350 ("igb") and couldn't.

#5

Updated by Adam Števko about 7 years ago

I am running OpenIndiana for multicast stream processing and the machine is connected to stacked Cisco switches via LACP trunk. The stream is processed in zones and is using vnics over aggregated interface.

Last week, one of the switches passed away and caused the bug to appear. The application, running in the zone, saw stream with double bit rate. After re-creating the vnics, the problem disappeared. The bug was seen on a machine with igb driver.

The same problem appeared some time ago on the machine with bnx driver.

I can post NIC specs if needed later this week.

#6

Updated by Adam Števko about 7 years ago

I am sorry for they delay, but I had some problems I had to solve first. I reproduced the problem and this is the stack from the misbehaving machine:

root@oi2:~# dtrace -n 'ip:::receive /args[2]->ip_saddr == "10.100.11.5"/ { stack(); }'
dtrace: description 'ip:::receive ' matched 5 probes
CPU     ID                    FUNCTION:NAME
 11   8406       ill_input_short_v4:receive 
              ip`ip_input+0x23b
              dls`i_dls_link_rx+0x2e7
              mac`mac_rx_deliver+0x5d
              mac`mac_rx_soft_ring_process+0x17a
              mac`mac_rx_srs_proto_fanout+0x4e5
              mac`mac_rx_srs_drain+0x26e
              mac`mac_rx_srs_process+0x1a9
              mac`mac_rx_classify+0x159
              mac`mac_rx_flow+0x54
              mac`mac_rx_common+0x1f6
              mac`mac_rx+0xac
              aggr`aggr_mac_rx+0x33
              aggr`aggr_recv_cb+0x18a
              mac`mac_promisc_dispatch_one+0x94
              mac`mac_promisc_dispatch+0x110
              mac`mac_rx_common+0x3e
              mac`mac_rx+0xac
              bnx`bnx_recv_ring_recv+0x3d9
              bnx`bnx_rxpkts_intr+0x20
              bnx`bnx_intr_recv+0x39

 11   8406       ill_input_short_v4:receive 
              ip`ip_input+0x23b
              dls`i_dls_link_rx+0x2e7
              mac`mac_rx_deliver+0x5d
              mac`mac_rx_soft_ring_process+0x17a
              mac`mac_rx_srs_proto_fanout+0x4e5
              mac`mac_rx_srs_drain+0x26e
              mac`mac_rx_srs_process+0x1a9
              mac`mac_rx_classify+0x159
              mac`mac_rx_flow+0x54
              mac`mac_rx_common+0x1f6
              mac`mac_rx+0xac
              aggr`aggr_mac_rx+0x33
              aggr`aggr_recv_cb+0x18a
              mac`mac_rx_deliver+0x5d
              mac`mac_rx_srs_drain+0x1c0
              mac`mac_rx_srs_process+0x1a9
              mac`mac_rx_common+0x16b
              mac`mac_rx+0xac
              bnx`bnx_recv_ring_recv+0x3d9
              bnx`bnx_rxpkts_intr+0x20

The stack from the correctly working machine looks like this (same stack appears when I remove/recreated vnics):

root@oi3:~#  dtrace -n 'ip:::receive /args[2]->ip_saddr == "10.100.11.4"/ { stack(); }'
dtrace: description 'ip:::receive ' matched 5 probes
CPU     ID                    FUNCTION:NAME
  0   1542       ill_input_short_v4:receive 
              ip`ip_input+0x23b
              dls`i_dls_link_rx+0x2e7
              mac`mac_rx_deliver+0x5d
              mac`mac_rx_soft_ring_process+0x17a
              mac`mac_rx_srs_proto_fanout+0x4e5
              mac`mac_rx_srs_drain+0x26e
              mac`mac_rx_srs_process+0x1a9
              mac`mac_rx_classify+0x159
              mac`mac_rx_flow+0x54
              mac`mac_rx_common+0x1f6
              mac`mac_rx+0xac
              aggr`aggr_mac_rx+0x33
              aggr`aggr_recv_cb+0x18a
              mac`mac_rx_deliver+0x5d
              mac`mac_rx_srs_drain+0x1c0
              mac`mac_rx_srs_process+0x1a9
              mac`mac_rx_common+0x16b
              mac`mac_rx+0xac
              bnx`bnx_recv_ring_recv+0x3d9
              bnx`bnx_rxpkts_intr+0x20

Both machines are using bnx drivers. When I destroy/recreatre vnics the problem goes away as expected.

Steps how to reproduce:
- create vnics
- reboot the machine
- ping the machine and ping will return DUPs

#7

Updated by Electric Monk 9 months ago

  • Status changed from New to Closed
  • % Done changed from 20 to 100

git commit 84de666edc7f7d835057ae4807a387447c086bcf

commit  84de666edc7f7d835057ae4807a387447c086bcf
Author: Ryan Zezeski <ryan@zinascii.com>
Date:   2020-03-02T07:24:09.000Z

    11490 SRS ring polling disabled for VLANs
    11491 Want DLS bypass for VLAN traffic
    11492 add VLVF bypass to ixgbe core
    2869 duplicate packets with vnics over aggrs
    11489 DLS stat delete and aggr kstat can deadlock
    Portions contributed by: Theo Schlossnagle <jesus@omniti.com>
    Reviewed by: Patrick Mooney <patrick.mooney@joyent.com>
    Reviewed by: Robert Mustacchi <rm@joyent.com>
    Reviewed by: Dan McDonald <danmcd@joyent.com>
    Reviewed by: Paul Winder <paul@winders.demon.co.uk>
    Approved by: Gordon Ross <gordon.w.ross@gmail.com>

Also available in: Atom PDF