Project

General

Profile

Bug #13230

i40e has duplicate traffic when used with bhyve/snoop running

Added by Jorge Schrauwen 6 months ago. Updated 6 days ago.

Status:
New
Priority:
Normal
Assignee:
Category:
driver - device drivers
Start date:
Due date:
% Done:

0%

Estimated time:
Difficulty:
Medium
Tags:
Gerrit CR:

Description

After having my board replaced my 2x 10G copper i40e nics are now working... sort of.

I noticed I was getting duplicate traffic inside bhyve which broke IPv6 and PPPoE, also duplicate traffic is not good regardless of what it breaks.

Initial tests:
- turn of offloading in the VM (had issues with this in the past): no change
- switch to a different underlying nic, igb in this case: works fine without dups
- switch back to i40e: dups are back...
- tried a 1G link instead of a 10G link: dups remained.

Left the system to run over igb (1G) instead of i40e (10G) until I had more time to debug.

Second set of tests:
- eliminated bhyve and add a vnic with a vlan on top of i40e: no dups (oh shit was it bhyve? ha no it wasn't)
- run snoop on the vnic: dups return (aha! so I guess whatever viona/bhyve is doing snoop also does)
( I think someone else was seeing a lot of re-transmits too with snoop running a few weeks ago, maybe related?)
- eliminated vnic with vlan tagging: same result! no dups unless snoop is running, then the dups are back.

Setup information:

[root@boron ~]# dladm show-phys
LINK         MEDIA                STATE      SPEED  DUPLEX    DEVICE
i40e0        Ethernet             up         1000   full      i40e0
igb0         Ethernet             up         1000   full      igb0
i40e1        Ethernet             up         1000   full      i40e1

i40e0: admin nictag (no vlan tagging)
igb0: trunk nictag (vlan tagging)
i40e1: what I want to use as trunk nictag, now just there for testing.

[root@boron ~]# dladm show-vnic
LINK         OVER       SPEED MACADDRESS        MACADDRTYPE VID  ZONE
net0         igb0       0     82:fc:c:c2:e4:6f  fixed       110  eceec5eb-22c7-e65b-fd6b-81ebdb36406b
net1         igb0       0     0:22:6:a:0:1      fixed       110  99e40ee7-a8f9-4b57-9225-e7bd19f64b07
net2         igb0       0     0:22:6:14:0:1     fixed       120  99e40ee7-a8f9-4b57-9225-e7bd19f64b07
net3         igb0       0     0:22:6:1e:0:1     fixed       130  99e40ee7-a8f9-4b57-9225-e7bd19f64b07
net4         igb0       0     0:22:6:28:0:1     fixed       140  99e40ee7-a8f9-4b57-9225-e7bd19f64b07
net0         igb0       0     0:22:6:5:0:a      fixed       10   99e40ee7-a8f9-4b57-9225-e7bd19f64b07
vnic110      i40e1      1000  2:8:20:43:c8:a0   random      110  --

vnic110, vnic on top of i40e1 tagged with vlan 110

[root@boron ~]# ipadm show-addr
ADDROBJ           TYPE     STATE        ADDR
lo0/v4            static   ok           127.0.0.1/8
i40e0/_a          static   ok           10.23.40.211/24
vnic110/dhcp      dhcp     ok           10.23.10.225/24
lo0/v6            static   ok           ::1/128
i40e0/?           static   ok           fe80::3eec:efff:fe0c:783a/10
i40e0/_b          static   ok           2a02:578:470f:40::211/64
i40e0/?           addrconf ok           2a02:578:470f:40:3eec:efff:fe0c:783a/64
i40e0/?           dhcp     ok           2a02:578:470f:40::246/128

10.23.40.211 gz-ip without vlan tagging
10.23.10.225 gz-ip with vlan tagging

Ping results without snoop running:

[iris :: sjorge][~]
[.]$ ping -c 10 10.23.40.211
PING 10.23.40.211 (10.23.40.211): 56 data bytes
64 bytes from 10.23.40.211: icmp_seq=0 ttl=255 time=0.745 ms
64 bytes from 10.23.40.211: icmp_seq=1 ttl=255 time=0.248 ms
64 bytes from 10.23.40.211: icmp_seq=2 ttl=255 time=0.234 ms
64 bytes from 10.23.40.211: icmp_seq=3 ttl=255 time=0.238 ms
64 bytes from 10.23.40.211: icmp_seq=4 ttl=255 time=0.349 ms
64 bytes from 10.23.40.211: icmp_seq=5 ttl=255 time=0.223 ms
64 bytes from 10.23.40.211: icmp_seq=6 ttl=255 time=0.224 ms
64 bytes from 10.23.40.211: icmp_seq=7 ttl=255 time=0.345 ms
64 bytes from 10.23.40.211: icmp_seq=8 ttl=255 time=0.339 ms
64 bytes from 10.23.40.211: icmp_seq=9 ttl=255 time=0.333 ms

--- 10.23.40.211 ping statistics ---
10 packets transmitted, 10 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 0.223/0.328/0.745/0.148 ms

[hyperon :: sjorge][~]
[■]$ ping -c 10 10.23.10.225
PING 10.23.10.225 (10.23.10.225): 56 data bytes
64 bytes from 10.23.10.225: icmp_seq=0 ttl=255 time=0.429 ms
64 bytes from 10.23.10.225: icmp_seq=1 ttl=255 time=0.420 ms
64 bytes from 10.23.10.225: icmp_seq=2 ttl=255 time=0.702 ms
64 bytes from 10.23.10.225: icmp_seq=3 ttl=255 time=0.523 ms
64 bytes from 10.23.10.225: icmp_seq=4 ttl=255 time=0.593 ms
64 bytes from 10.23.10.225: icmp_seq=5 ttl=255 time=0.593 ms
64 bytes from 10.23.10.225: icmp_seq=6 ttl=255 time=0.605 ms
64 bytes from 10.23.10.225: icmp_seq=7 ttl=255 time=0.575 ms
64 bytes from 10.23.10.225: icmp_seq=8 ttl=255 time=0.396 ms
64 bytes from 10.23.10.225: icmp_seq=9 ttl=255 time=0.683 ms

--- 10.23.10.225 ping statistics ---
10 packets transmitted, 10 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 0.396/0.552/0.702/0.102 ms

Pings were done from macOS for 10.23.10.0/24 and from a freebsd host for 10.23.40.0/24... to make sure there was no router in play.

Ping results with snoop running:

[hyperon :: sjorge][~]
[■]$ ping -c 10 10.23.10.225
PING 10.23.10.225 (10.23.10.225): 56 data bytes
64 bytes from 10.23.10.225: icmp_seq=0 ttl=255 time=0.575 ms
64 bytes from 10.23.10.225: icmp_seq=0 ttl=255 time=0.580 ms (DUP!)
64 bytes from 10.23.10.225: icmp_seq=1 ttl=255 time=8.708 ms
64 bytes from 10.23.10.225: icmp_seq=1 ttl=255 time=8.711 ms (DUP!)
64 bytes from 10.23.10.225: icmp_seq=2 ttl=255 time=0.447 ms
64 bytes from 10.23.10.225: icmp_seq=2 ttl=255 time=0.452 ms (DUP!)
64 bytes from 10.23.10.225: icmp_seq=3 ttl=255 time=0.446 ms
64 bytes from 10.23.10.225: icmp_seq=3 ttl=255 time=0.451 ms (DUP!)
64 bytes from 10.23.10.225: icmp_seq=4 ttl=255 time=0.389 ms
64 bytes from 10.23.10.225: icmp_seq=4 ttl=255 time=0.436 ms (DUP!)
64 bytes from 10.23.10.225: icmp_seq=5 ttl=255 time=0.589 ms
64 bytes from 10.23.10.225: icmp_seq=5 ttl=255 time=0.597 ms (DUP!)
64 bytes from 10.23.10.225: icmp_seq=6 ttl=255 time=0.601 ms
64 bytes from 10.23.10.225: icmp_seq=6 ttl=255 time=0.613 ms (DUP!)
64 bytes from 10.23.10.225: icmp_seq=7 ttl=255 time=0.398 ms
64 bytes from 10.23.10.225: icmp_seq=7 ttl=255 time=0.611 ms (DUP!)
64 bytes from 10.23.10.225: icmp_seq=8 ttl=255 time=0.586 ms
64 bytes from 10.23.10.225: icmp_seq=8 ttl=255 time=0.622 ms (DUP!)
64 bytes from 10.23.10.225: icmp_seq=9 ttl=255 time=8.514 ms

--- 10.23.10.225 ping statistics ---
10 packets transmitted, 10 packets received, +9 duplicates, 0.0% packet loss
round-trip min/avg/max/stddev = 0.389/1.807/8.711/2.962 ms

[root@boron ~]# snoop -d vnic110 'icmp'
Using device vnic110 (promiscuous mode)
hyperon.vlan10.acheron.be -> 10.23.10.225 ICMP Echo request (ID: 50479 Sequence number: 0)
hyperon.vlan10.acheron.be -> 10.23.10.225 ICMP Echo request (ID: 50479 Sequence number: 0)
10.23.10.225 -> hyperon.vlan10.acheron.be ICMP Echo reply (ID: 50479 Sequence number: 0)
10.23.10.225 -> hyperon.vlan10.acheron.be ICMP Echo reply (ID: 50479 Sequence number: 0)
hyperon.vlan10.acheron.be -> 10.23.10.225 ICMP Echo request (ID: 50479 Sequence number: 1)
hyperon.vlan10.acheron.be -> 10.23.10.225 ICMP Echo request (ID: 50479 Sequence number: 1)
10.23.10.225 -> hyperon.vlan10.acheron.be ICMP Echo reply (ID: 50479 Sequence number: 1)
10.23.10.225 -> hyperon.vlan10.acheron.be ICMP Echo reply (ID: 50479 Sequence number: 1)
hyperon.vlan10.acheron.be -> 10.23.10.225 ICMP Echo request (ID: 50479 Sequence number: 2)
10.23.10.225 -> hyperon.vlan10.acheron.be ICMP Echo reply (ID: 50479 Sequence number: 2)
hyperon.vlan10.acheron.be -> 10.23.10.225 ICMP Echo request (ID: 50479 Sequence number: 2)
10.23.10.225 -> hyperon.vlan10.acheron.be ICMP Echo reply (ID: 50479 Sequence number: 2)
hyperon.vlan10.acheron.be -> 10.23.10.225 ICMP Echo request (ID: 50479 Sequence number: 3)
hyperon.vlan10.acheron.be -> 10.23.10.225 ICMP Echo request (ID: 50479 Sequence number: 3)
10.23.10.225 -> hyperon.vlan10.acheron.be ICMP Echo reply (ID: 50479 Sequence number: 3)
10.23.10.225 -> hyperon.vlan10.acheron.be ICMP Echo reply (ID: 50479 Sequence number: 3)
hyperon.vlan10.acheron.be -> 10.23.10.225 ICMP Echo request (ID: 50479 Sequence number: 4)
hyperon.vlan10.acheron.be -> 10.23.10.225 ICMP Echo request (ID: 50479 Sequence number: 4)
10.23.10.225 -> hyperon.vlan10.acheron.be ICMP Echo reply (ID: 50479 Sequence number: 4)
10.23.10.225 -> hyperon.vlan10.acheron.be ICMP Echo reply (ID: 50479 Sequence number: 4)
hyperon.vlan10.acheron.be -> 10.23.10.225 ICMP Echo request (ID: 50479 Sequence number: 5)
hyperon.vlan10.acheron.be -> 10.23.10.225 ICMP Echo request (ID: 50479 Sequence number: 5)
10.23.10.225 -> hyperon.vlan10.acheron.be ICMP Echo reply (ID: 50479 Sequence number: 5)
10.23.10.225 -> hyperon.vlan10.acheron.be ICMP Echo reply (ID: 50479 Sequence number: 5)
hyperon.vlan10.acheron.be -> 10.23.10.225 ICMP Echo request (ID: 50479 Sequence number: 6)
hyperon.vlan10.acheron.be -> 10.23.10.225 ICMP Echo request (ID: 50479 Sequence number: 6)
10.23.10.225 -> hyperon.vlan10.acheron.be ICMP Echo reply (ID: 50479 Sequence number: 6)
10.23.10.225 -> hyperon.vlan10.acheron.be ICMP Echo reply (ID: 50479 Sequence number: 6)
hyperon.vlan10.acheron.be -> 10.23.10.225 ICMP Echo request (ID: 50479 Sequence number: 7)
hyperon.vlan10.acheron.be -> 10.23.10.225 ICMP Echo request (ID: 50479 Sequence number: 7)
10.23.10.225 -> hyperon.vlan10.acheron.be ICMP Echo reply (ID: 50479 Sequence number: 7)
10.23.10.225 -> hyperon.vlan10.acheron.be ICMP Echo reply (ID: 50479 Sequence number: 7)
hyperon.vlan10.acheron.be -> 10.23.10.225 ICMP Echo request (ID: 50479 Sequence number: 8)
10.23.10.225 -> hyperon.vlan10.acheron.be ICMP Echo reply (ID: 50479 Sequence number: 8)
hyperon.vlan10.acheron.be -> 10.23.10.225 ICMP Echo request (ID: 50479 Sequence number: 8)
10.23.10.225 -> hyperon.vlan10.acheron.be ICMP Echo reply (ID: 50479 Sequence number: 8)
hyperon.vlan10.acheron.be -> 10.23.10.225 ICMP Echo request (ID: 50479 Sequence number: 9)
hyperon.vlan10.acheron.be -> 10.23.10.225 ICMP Echo request (ID: 50479 Sequence number: 9)
10.23.10.225 -> hyperon.vlan10.acheron.be ICMP Echo reply (ID: 50479 Sequence number: 9)
10.23.10.225 -> hyperon.vlan10.acheron.be ICMP Echo reply (ID: 50479 Sequence number: 9)

[iris :: sjorge][~]
[.]$ ping -c 10 10.23.40.211
PING 10.23.40.211 (10.23.40.211): 56 data bytes
64 bytes from 10.23.40.211: icmp_seq=0 ttl=255 time=0.524 ms
64 bytes from 10.23.40.211: icmp_seq=0 ttl=255 time=0.633 ms (DUP!)
64 bytes from 10.23.40.211: icmp_seq=1 ttl=255 time=0.402 ms
64 bytes from 10.23.40.211: icmp_seq=1 ttl=255 time=0.519 ms (DUP!)
64 bytes from 10.23.40.211: icmp_seq=2 ttl=255 time=0.391 ms
64 bytes from 10.23.40.211: icmp_seq=2 ttl=255 time=0.518 ms (DUP!)
64 bytes from 10.23.40.211: icmp_seq=3 ttl=255 time=0.407 ms
64 bytes from 10.23.40.211: icmp_seq=3 ttl=255 time=0.535 ms (DUP!)
64 bytes from 10.23.40.211: icmp_seq=4 ttl=255 time=0.406 ms
64 bytes from 10.23.40.211: icmp_seq=4 ttl=255 time=0.427 ms (DUP!)
64 bytes from 10.23.40.211: icmp_seq=5 ttl=255 time=0.400 ms
64 bytes from 10.23.40.211: icmp_seq=5 ttl=255 time=0.421 ms (DUP!)
64 bytes from 10.23.40.211: icmp_seq=6 ttl=255 time=0.402 ms
64 bytes from 10.23.40.211: icmp_seq=6 ttl=255 time=0.424 ms (DUP!)
64 bytes from 10.23.40.211: icmp_seq=7 ttl=255 time=0.528 ms
64 bytes from 10.23.40.211: icmp_seq=7 ttl=255 time=0.550 ms (DUP!)
64 bytes from 10.23.40.211: icmp_seq=8 ttl=255 time=0.516 ms
64 bytes from 10.23.40.211: icmp_seq=8 ttl=255 time=0.536 ms (DUP!)
64 bytes from 10.23.40.211: icmp_seq=9 ttl=255 time=0.515 ms

--- 10.23.40.211 ping statistics ---
10 packets transmitted, 10 packets received, +9 duplicates, 0.0% packet loss
round-trip min/avg/max/stddev = 0.391/0.477/0.633/0.069 ms

[root@boron ~]# snoop -d i40e0 'icmp'
Using device i40e0 (promiscuous mode)
iris.vlan40.acheron.be -> boron        ICMP Echo request (ID: 7992 Sequence number: 0)
iris.vlan40.acheron.be -> boron        ICMP Echo request (ID: 7992 Sequence number: 0)
       boron -> iris.vlan40.acheron.be ICMP Echo reply (ID: 7992 Sequence number: 0)
       boron -> iris.vlan40.acheron.be ICMP Echo reply (ID: 7992 Sequence number: 0)
iris.vlan40.acheron.be -> boron        ICMP Echo request (ID: 7992 Sequence number: 1)
       boron -> iris.vlan40.acheron.be ICMP Echo reply (ID: 7992 Sequence number: 1)
iris.vlan40.acheron.be -> boron        ICMP Echo request (ID: 7992 Sequence number: 1)
       boron -> iris.vlan40.acheron.be ICMP Echo reply (ID: 7992 Sequence number: 1)
iris.vlan40.acheron.be -> boron        ICMP Echo request (ID: 7992 Sequence number: 2)
       boron -> iris.vlan40.acheron.be ICMP Echo reply (ID: 7992 Sequence number: 2)
iris.vlan40.acheron.be -> boron        ICMP Echo request (ID: 7992 Sequence number: 2)
       boron -> iris.vlan40.acheron.be ICMP Echo reply (ID: 7992 Sequence number: 2)
iris.vlan40.acheron.be -> boron        ICMP Echo request (ID: 7992 Sequence number: 3)
       boron -> iris.vlan40.acheron.be ICMP Echo reply (ID: 7992 Sequence number: 3)
iris.vlan40.acheron.be -> boron        ICMP Echo request (ID: 7992 Sequence number: 3)
       boron -> iris.vlan40.acheron.be ICMP Echo reply (ID: 7992 Sequence number: 3)
iris.vlan40.acheron.be -> boron        ICMP Echo request (ID: 7992 Sequence number: 4)
iris.vlan40.acheron.be -> boron        ICMP Echo request (ID: 7992 Sequence number: 4)
       boron -> iris.vlan40.acheron.be ICMP Echo reply (ID: 7992 Sequence number: 4)
       boron -> iris.vlan40.acheron.be ICMP Echo reply (ID: 7992 Sequence number: 4)
iris.vlan40.acheron.be -> boron        ICMP Echo request (ID: 7992 Sequence number: 5)
iris.vlan40.acheron.be -> boron        ICMP Echo request (ID: 7992 Sequence number: 5)
       boron -> iris.vlan40.acheron.be ICMP Echo reply (ID: 7992 Sequence number: 5)
       boron -> iris.vlan40.acheron.be ICMP Echo reply (ID: 7992 Sequence number: 5)
iris.vlan40.acheron.be -> boron        ICMP Echo request (ID: 7992 Sequence number: 6)
iris.vlan40.acheron.be -> boron        ICMP Echo request (ID: 7992 Sequence number: 6)
       boron -> iris.vlan40.acheron.be ICMP Echo reply (ID: 7992 Sequence number: 6)
       boron -> iris.vlan40.acheron.be ICMP Echo reply (ID: 7992 Sequence number: 6)
iris.vlan40.acheron.be -> boron        ICMP Echo request (ID: 7992 Sequence number: 7)
iris.vlan40.acheron.be -> boron        ICMP Echo request (ID: 7992 Sequence number: 7)
       boron -> iris.vlan40.acheron.be ICMP Echo reply (ID: 7992 Sequence number: 7)
       boron -> iris.vlan40.acheron.be ICMP Echo reply (ID: 7992 Sequence number: 7)
iris.vlan40.acheron.be -> boron        ICMP Echo request (ID: 7992 Sequence number: 8)
iris.vlan40.acheron.be -> boron        ICMP Echo request (ID: 7992 Sequence number: 8)
       boron -> iris.vlan40.acheron.be ICMP Echo reply (ID: 7992 Sequence number: 8)
       boron -> iris.vlan40.acheron.be ICMP Echo reply (ID: 7992 Sequence number: 8)
iris.vlan40.acheron.be -> boron        ICMP Echo request (ID: 7992 Sequence number: 9)
iris.vlan40.acheron.be -> boron        ICMP Echo request (ID: 7992 Sequence number: 9)
       boron -> iris.vlan40.acheron.be ICMP Echo reply (ID: 7992 Sequence number: 9)
       boron -> iris.vlan40.acheron.be ICMP Echo reply (ID: 7992 Sequence number: 9)

Also available in: Atom PDF