Bug #2869
closed
duplicate packets with vnics over aggrs
100%
Description
On a current system with two e1000g NICs in an LACP aggregate, on which vnics are placed, we see duplicate packets. (simple ping will show DUP responses).
Repeat reports in igb are here: http://blogs.everycity.co.uk/alasdair/2010/10/weird-opensolariscrossbow-issue-with-aggregations-and-vlansvnics/
and bge here: http://mail.opensolaris.org/pipermail/crossbow-discuss/2009-June/001923.html
A user claims it was fixed in Solaris 11 GA (not that it helps).
Updated by Rich Lowe over 11 years ago
- Subject changed from duplicate packets to duplicate packets with vnics over aggrs
Updated by Theo Schlossnagle over 11 years ago
- % Done changed from 0 to 20
The issue appears to fall along the lines of hardware support.
If the underlying ethernet devices support hardware MAC matching, everything works. When a device is flipped into promiscuous mode, packets are delivered via mac_promisc_dispatch and then not delivered by the hardware matching code path.
On the other hand, in software mode, we fallback to pushing packets through flows. However, there is no way for the software flow code to know whether it should be pushing the packets in (as that in an impl detail).
On our systems with e1000g devices, everything works fine as hardware MAC matching is supported and the aggr never flips the underlying interfaces into promiscuous mode. On our nge-based machines, aggr will set the underlying interfaces into promiscuous mode an a packet wil be delivered twice in mac_rx_common via mac_promisc_dispatch and then again via (flent->fe_cb_fn)(flent->fe_cb_arg1, flent->fe_cb_arg2, mp_chain, B_FALSE);
I propose making the aggr code aware of this and avoiding it at that layer. Only the aggr layer knows whether it has put the interfaces into promiscuous mode for "ordinary" purposes.
Updated by Theo Schlossnagle over 11 years ago
For reference, here is a proposed fix: http://cr.illumos.org/~webrev/postwait/2869/
This fix is not optimal as the mac layer is still producing two chains, one of which we are freeing.
It is possible that mac_rx_set and mac_rx_clear are malfunctioning and the aggr is not correctly registering itself/deregistering itself, but I think the interplay with VNICs atop them makes that unclear.
I've been unable to diagnose malfunctioning in the MAC layer and have not been successful removing the production of the duplicate chain of packets. (the above fix simply suppresses them before they are delivered through the aggr)
Updated by Dan McDonald about 10 years ago
Previously, it was stated that this bug manifests with devices in the aggr that don't support MAC matching in hardware.
Assuming so, is there a whitelist of HW where this problem does not occur? I just tried to reproduce this with Intel I350 ("igb") and couldn't.
Updated by Adam Števko almost 10 years ago
I am running OpenIndiana for multicast stream processing and the machine is connected to stacked Cisco switches via LACP trunk. The stream is processed in zones and is using vnics over aggregated interface.
Last week, one of the switches passed away and caused the bug to appear. The application, running in the zone, saw stream with double bit rate. After re-creating the vnics, the problem disappeared. The bug was seen on a machine with igb driver.
The same problem appeared some time ago on the machine with bnx driver.
I can post NIC specs if needed later this week.
Updated by Adam Števko almost 10 years ago
I am sorry for they delay, but I had some problems I had to solve first. I reproduced the problem and this is the stack from the misbehaving machine:
root@oi2:~# dtrace -n 'ip:::receive /args[2]->ip_saddr == "10.100.11.5"/ { stack(); }'
dtrace: description 'ip:::receive ' matched 5 probes
CPU ID FUNCTION:NAME
11 8406 ill_input_short_v4:receive
ip`ip_input+0x23b
dls`i_dls_link_rx+0x2e7
mac`mac_rx_deliver+0x5d
mac`mac_rx_soft_ring_process+0x17a
mac`mac_rx_srs_proto_fanout+0x4e5
mac`mac_rx_srs_drain+0x26e
mac`mac_rx_srs_process+0x1a9
mac`mac_rx_classify+0x159
mac`mac_rx_flow+0x54
mac`mac_rx_common+0x1f6
mac`mac_rx+0xac
aggr`aggr_mac_rx+0x33
aggr`aggr_recv_cb+0x18a
mac`mac_promisc_dispatch_one+0x94
mac`mac_promisc_dispatch+0x110
mac`mac_rx_common+0x3e
mac`mac_rx+0xac
bnx`bnx_recv_ring_recv+0x3d9
bnx`bnx_rxpkts_intr+0x20
bnx`bnx_intr_recv+0x39
11 8406 ill_input_short_v4:receive
ip`ip_input+0x23b
dls`i_dls_link_rx+0x2e7
mac`mac_rx_deliver+0x5d
mac`mac_rx_soft_ring_process+0x17a
mac`mac_rx_srs_proto_fanout+0x4e5
mac`mac_rx_srs_drain+0x26e
mac`mac_rx_srs_process+0x1a9
mac`mac_rx_classify+0x159
mac`mac_rx_flow+0x54
mac`mac_rx_common+0x1f6
mac`mac_rx+0xac
aggr`aggr_mac_rx+0x33
aggr`aggr_recv_cb+0x18a
mac`mac_rx_deliver+0x5d
mac`mac_rx_srs_drain+0x1c0
mac`mac_rx_srs_process+0x1a9
mac`mac_rx_common+0x16b
mac`mac_rx+0xac
bnx`bnx_recv_ring_recv+0x3d9
bnx`bnx_rxpkts_intr+0x20
The stack from the correctly working machine looks like this (same stack appears when I remove/recreated vnics):
root@oi3:~# dtrace -n 'ip:::receive /args[2]->ip_saddr == "10.100.11.4"/ { stack(); }'
dtrace: description 'ip:::receive ' matched 5 probes
CPU ID FUNCTION:NAME
0 1542 ill_input_short_v4:receive
ip`ip_input+0x23b
dls`i_dls_link_rx+0x2e7
mac`mac_rx_deliver+0x5d
mac`mac_rx_soft_ring_process+0x17a
mac`mac_rx_srs_proto_fanout+0x4e5
mac`mac_rx_srs_drain+0x26e
mac`mac_rx_srs_process+0x1a9
mac`mac_rx_classify+0x159
mac`mac_rx_flow+0x54
mac`mac_rx_common+0x1f6
mac`mac_rx+0xac
aggr`aggr_mac_rx+0x33
aggr`aggr_recv_cb+0x18a
mac`mac_rx_deliver+0x5d
mac`mac_rx_srs_drain+0x1c0
mac`mac_rx_srs_process+0x1a9
mac`mac_rx_common+0x16b
mac`mac_rx+0xac
bnx`bnx_recv_ring_recv+0x3d9
bnx`bnx_rxpkts_intr+0x20
Both machines are using bnx drivers. When I destroy/recreatre vnics the problem goes away as expected.
Steps how to reproduce:
- create vnics
- reboot the machine
- ping the machine and ping will return DUPs
Updated by Electric Monk over 3 years ago
- Status changed from New to Closed
- % Done changed from 20 to 100
git commit 84de666edc7f7d835057ae4807a387447c086bcf
commit 84de666edc7f7d835057ae4807a387447c086bcf
Author: Ryan Zezeski <ryan@zinascii.com>
Date: 2020-03-02T07:24:09.000Z
11490 SRS ring polling disabled for VLANs
11491 Want DLS bypass for VLAN traffic
11492 add VLVF bypass to ixgbe core
2869 duplicate packets with vnics over aggrs
11489 DLS stat delete and aggr kstat can deadlock
Portions contributed by: Theo Schlossnagle <jesus@omniti.com>
Reviewed by: Patrick Mooney <patrick.mooney@joyent.com>
Reviewed by: Robert Mustacchi <rm@joyent.com>
Reviewed by: Dan McDonald <danmcd@joyent.com>
Reviewed by: Paul Winder <paul@winders.demon.co.uk>
Approved by: Gordon Ross <gordon.w.ross@gmail.com>