Bug #13230
i40e has duplicate traffic when used with bhyve/snoop running
Start date:
Due date:
% Done:
0%
Estimated time:
Difficulty:
Medium
Tags:
Gerrit CR:
Description
After having my board replaced my 2x 10G copper i40e nics are now working... sort of.
I noticed I was getting duplicate traffic inside bhyve which broke IPv6 and PPPoE, also duplicate traffic is not good regardless of what it breaks.
Initial tests:
- turn of offloading in the VM (had issues with this in the past): no change
- switch to a different underlying nic, igb in this case: works fine without dups
- switch back to i40e: dups are back...
- tried a 1G link instead of a 10G link: dups remained.
Left the system to run over igb (1G) instead of i40e (10G) until I had more time to debug.
Second set of tests:
- eliminated bhyve and add a vnic with a vlan on top of i40e: no dups (oh shit was it bhyve? ha no it wasn't)
- run snoop on the vnic: dups return (aha! so I guess whatever viona/bhyve is doing snoop also does)
( I think someone else was seeing a lot of re-transmits too with snoop running a few weeks ago, maybe related?)
- eliminated vnic with vlan tagging: same result! no dups unless snoop is running, then the dups are back.
Setup information:
[root@boron ~]# dladm show-phys LINK MEDIA STATE SPEED DUPLEX DEVICE i40e0 Ethernet up 1000 full i40e0 igb0 Ethernet up 1000 full igb0 i40e1 Ethernet up 1000 full i40e1
i40e0: admin nictag (no vlan tagging)
igb0: trunk nictag (vlan tagging)
i40e1: what I want to use as trunk nictag, now just there for testing.
[root@boron ~]# dladm show-vnic LINK OVER SPEED MACADDRESS MACADDRTYPE VID ZONE net0 igb0 0 82:fc:c:c2:e4:6f fixed 110 eceec5eb-22c7-e65b-fd6b-81ebdb36406b net1 igb0 0 0:22:6:a:0:1 fixed 110 99e40ee7-a8f9-4b57-9225-e7bd19f64b07 net2 igb0 0 0:22:6:14:0:1 fixed 120 99e40ee7-a8f9-4b57-9225-e7bd19f64b07 net3 igb0 0 0:22:6:1e:0:1 fixed 130 99e40ee7-a8f9-4b57-9225-e7bd19f64b07 net4 igb0 0 0:22:6:28:0:1 fixed 140 99e40ee7-a8f9-4b57-9225-e7bd19f64b07 net0 igb0 0 0:22:6:5:0:a fixed 10 99e40ee7-a8f9-4b57-9225-e7bd19f64b07 vnic110 i40e1 1000 2:8:20:43:c8:a0 random 110 --
vnic110, vnic on top of i40e1 tagged with vlan 110
[root@boron ~]# ipadm show-addr ADDROBJ TYPE STATE ADDR lo0/v4 static ok 127.0.0.1/8 i40e0/_a static ok 10.23.40.211/24 vnic110/dhcp dhcp ok 10.23.10.225/24 lo0/v6 static ok ::1/128 i40e0/? static ok fe80::3eec:efff:fe0c:783a/10 i40e0/_b static ok 2a02:578:470f:40::211/64 i40e0/? addrconf ok 2a02:578:470f:40:3eec:efff:fe0c:783a/64 i40e0/? dhcp ok 2a02:578:470f:40::246/128
10.23.40.211 gz-ip without vlan tagging
10.23.10.225 gz-ip with vlan tagging
Ping results without snoop running:
[iris :: sjorge][~] [.]$ ping -c 10 10.23.40.211 PING 10.23.40.211 (10.23.40.211): 56 data bytes 64 bytes from 10.23.40.211: icmp_seq=0 ttl=255 time=0.745 ms 64 bytes from 10.23.40.211: icmp_seq=1 ttl=255 time=0.248 ms 64 bytes from 10.23.40.211: icmp_seq=2 ttl=255 time=0.234 ms 64 bytes from 10.23.40.211: icmp_seq=3 ttl=255 time=0.238 ms 64 bytes from 10.23.40.211: icmp_seq=4 ttl=255 time=0.349 ms 64 bytes from 10.23.40.211: icmp_seq=5 ttl=255 time=0.223 ms 64 bytes from 10.23.40.211: icmp_seq=6 ttl=255 time=0.224 ms 64 bytes from 10.23.40.211: icmp_seq=7 ttl=255 time=0.345 ms 64 bytes from 10.23.40.211: icmp_seq=8 ttl=255 time=0.339 ms 64 bytes from 10.23.40.211: icmp_seq=9 ttl=255 time=0.333 ms --- 10.23.40.211 ping statistics --- 10 packets transmitted, 10 packets received, 0.0% packet loss round-trip min/avg/max/stddev = 0.223/0.328/0.745/0.148 ms [hyperon :: sjorge][~] [■]$ ping -c 10 10.23.10.225 PING 10.23.10.225 (10.23.10.225): 56 data bytes 64 bytes from 10.23.10.225: icmp_seq=0 ttl=255 time=0.429 ms 64 bytes from 10.23.10.225: icmp_seq=1 ttl=255 time=0.420 ms 64 bytes from 10.23.10.225: icmp_seq=2 ttl=255 time=0.702 ms 64 bytes from 10.23.10.225: icmp_seq=3 ttl=255 time=0.523 ms 64 bytes from 10.23.10.225: icmp_seq=4 ttl=255 time=0.593 ms 64 bytes from 10.23.10.225: icmp_seq=5 ttl=255 time=0.593 ms 64 bytes from 10.23.10.225: icmp_seq=6 ttl=255 time=0.605 ms 64 bytes from 10.23.10.225: icmp_seq=7 ttl=255 time=0.575 ms 64 bytes from 10.23.10.225: icmp_seq=8 ttl=255 time=0.396 ms 64 bytes from 10.23.10.225: icmp_seq=9 ttl=255 time=0.683 ms --- 10.23.10.225 ping statistics --- 10 packets transmitted, 10 packets received, 0.0% packet loss round-trip min/avg/max/stddev = 0.396/0.552/0.702/0.102 ms
Pings were done from macOS for 10.23.10.0/24 and from a freebsd host for 10.23.40.0/24... to make sure there was no router in play.
Ping results with snoop running:
[hyperon :: sjorge][~] [■]$ ping -c 10 10.23.10.225 PING 10.23.10.225 (10.23.10.225): 56 data bytes 64 bytes from 10.23.10.225: icmp_seq=0 ttl=255 time=0.575 ms 64 bytes from 10.23.10.225: icmp_seq=0 ttl=255 time=0.580 ms (DUP!) 64 bytes from 10.23.10.225: icmp_seq=1 ttl=255 time=8.708 ms 64 bytes from 10.23.10.225: icmp_seq=1 ttl=255 time=8.711 ms (DUP!) 64 bytes from 10.23.10.225: icmp_seq=2 ttl=255 time=0.447 ms 64 bytes from 10.23.10.225: icmp_seq=2 ttl=255 time=0.452 ms (DUP!) 64 bytes from 10.23.10.225: icmp_seq=3 ttl=255 time=0.446 ms 64 bytes from 10.23.10.225: icmp_seq=3 ttl=255 time=0.451 ms (DUP!) 64 bytes from 10.23.10.225: icmp_seq=4 ttl=255 time=0.389 ms 64 bytes from 10.23.10.225: icmp_seq=4 ttl=255 time=0.436 ms (DUP!) 64 bytes from 10.23.10.225: icmp_seq=5 ttl=255 time=0.589 ms 64 bytes from 10.23.10.225: icmp_seq=5 ttl=255 time=0.597 ms (DUP!) 64 bytes from 10.23.10.225: icmp_seq=6 ttl=255 time=0.601 ms 64 bytes from 10.23.10.225: icmp_seq=6 ttl=255 time=0.613 ms (DUP!) 64 bytes from 10.23.10.225: icmp_seq=7 ttl=255 time=0.398 ms 64 bytes from 10.23.10.225: icmp_seq=7 ttl=255 time=0.611 ms (DUP!) 64 bytes from 10.23.10.225: icmp_seq=8 ttl=255 time=0.586 ms 64 bytes from 10.23.10.225: icmp_seq=8 ttl=255 time=0.622 ms (DUP!) 64 bytes from 10.23.10.225: icmp_seq=9 ttl=255 time=8.514 ms --- 10.23.10.225 ping statistics --- 10 packets transmitted, 10 packets received, +9 duplicates, 0.0% packet loss round-trip min/avg/max/stddev = 0.389/1.807/8.711/2.962 ms [root@boron ~]# snoop -d vnic110 'icmp' Using device vnic110 (promiscuous mode) hyperon.vlan10.acheron.be -> 10.23.10.225 ICMP Echo request (ID: 50479 Sequence number: 0) hyperon.vlan10.acheron.be -> 10.23.10.225 ICMP Echo request (ID: 50479 Sequence number: 0) 10.23.10.225 -> hyperon.vlan10.acheron.be ICMP Echo reply (ID: 50479 Sequence number: 0) 10.23.10.225 -> hyperon.vlan10.acheron.be ICMP Echo reply (ID: 50479 Sequence number: 0) hyperon.vlan10.acheron.be -> 10.23.10.225 ICMP Echo request (ID: 50479 Sequence number: 1) hyperon.vlan10.acheron.be -> 10.23.10.225 ICMP Echo request (ID: 50479 Sequence number: 1) 10.23.10.225 -> hyperon.vlan10.acheron.be ICMP Echo reply (ID: 50479 Sequence number: 1) 10.23.10.225 -> hyperon.vlan10.acheron.be ICMP Echo reply (ID: 50479 Sequence number: 1) hyperon.vlan10.acheron.be -> 10.23.10.225 ICMP Echo request (ID: 50479 Sequence number: 2) 10.23.10.225 -> hyperon.vlan10.acheron.be ICMP Echo reply (ID: 50479 Sequence number: 2) hyperon.vlan10.acheron.be -> 10.23.10.225 ICMP Echo request (ID: 50479 Sequence number: 2) 10.23.10.225 -> hyperon.vlan10.acheron.be ICMP Echo reply (ID: 50479 Sequence number: 2) hyperon.vlan10.acheron.be -> 10.23.10.225 ICMP Echo request (ID: 50479 Sequence number: 3) hyperon.vlan10.acheron.be -> 10.23.10.225 ICMP Echo request (ID: 50479 Sequence number: 3) 10.23.10.225 -> hyperon.vlan10.acheron.be ICMP Echo reply (ID: 50479 Sequence number: 3) 10.23.10.225 -> hyperon.vlan10.acheron.be ICMP Echo reply (ID: 50479 Sequence number: 3) hyperon.vlan10.acheron.be -> 10.23.10.225 ICMP Echo request (ID: 50479 Sequence number: 4) hyperon.vlan10.acheron.be -> 10.23.10.225 ICMP Echo request (ID: 50479 Sequence number: 4) 10.23.10.225 -> hyperon.vlan10.acheron.be ICMP Echo reply (ID: 50479 Sequence number: 4) 10.23.10.225 -> hyperon.vlan10.acheron.be ICMP Echo reply (ID: 50479 Sequence number: 4) hyperon.vlan10.acheron.be -> 10.23.10.225 ICMP Echo request (ID: 50479 Sequence number: 5) hyperon.vlan10.acheron.be -> 10.23.10.225 ICMP Echo request (ID: 50479 Sequence number: 5) 10.23.10.225 -> hyperon.vlan10.acheron.be ICMP Echo reply (ID: 50479 Sequence number: 5) 10.23.10.225 -> hyperon.vlan10.acheron.be ICMP Echo reply (ID: 50479 Sequence number: 5) hyperon.vlan10.acheron.be -> 10.23.10.225 ICMP Echo request (ID: 50479 Sequence number: 6) hyperon.vlan10.acheron.be -> 10.23.10.225 ICMP Echo request (ID: 50479 Sequence number: 6) 10.23.10.225 -> hyperon.vlan10.acheron.be ICMP Echo reply (ID: 50479 Sequence number: 6) 10.23.10.225 -> hyperon.vlan10.acheron.be ICMP Echo reply (ID: 50479 Sequence number: 6) hyperon.vlan10.acheron.be -> 10.23.10.225 ICMP Echo request (ID: 50479 Sequence number: 7) hyperon.vlan10.acheron.be -> 10.23.10.225 ICMP Echo request (ID: 50479 Sequence number: 7) 10.23.10.225 -> hyperon.vlan10.acheron.be ICMP Echo reply (ID: 50479 Sequence number: 7) 10.23.10.225 -> hyperon.vlan10.acheron.be ICMP Echo reply (ID: 50479 Sequence number: 7) hyperon.vlan10.acheron.be -> 10.23.10.225 ICMP Echo request (ID: 50479 Sequence number: 8) 10.23.10.225 -> hyperon.vlan10.acheron.be ICMP Echo reply (ID: 50479 Sequence number: 8) hyperon.vlan10.acheron.be -> 10.23.10.225 ICMP Echo request (ID: 50479 Sequence number: 8) 10.23.10.225 -> hyperon.vlan10.acheron.be ICMP Echo reply (ID: 50479 Sequence number: 8) hyperon.vlan10.acheron.be -> 10.23.10.225 ICMP Echo request (ID: 50479 Sequence number: 9) hyperon.vlan10.acheron.be -> 10.23.10.225 ICMP Echo request (ID: 50479 Sequence number: 9) 10.23.10.225 -> hyperon.vlan10.acheron.be ICMP Echo reply (ID: 50479 Sequence number: 9) 10.23.10.225 -> hyperon.vlan10.acheron.be ICMP Echo reply (ID: 50479 Sequence number: 9)
[iris :: sjorge][~]
[.]$ ping -c 10 10.23.40.211
PING 10.23.40.211 (10.23.40.211): 56 data bytes
64 bytes from 10.23.40.211: icmp_seq=0 ttl=255 time=0.524 ms
64 bytes from 10.23.40.211: icmp_seq=0 ttl=255 time=0.633 ms (DUP!)
64 bytes from 10.23.40.211: icmp_seq=1 ttl=255 time=0.402 ms
64 bytes from 10.23.40.211: icmp_seq=1 ttl=255 time=0.519 ms (DUP!)
64 bytes from 10.23.40.211: icmp_seq=2 ttl=255 time=0.391 ms
64 bytes from 10.23.40.211: icmp_seq=2 ttl=255 time=0.518 ms (DUP!)
64 bytes from 10.23.40.211: icmp_seq=3 ttl=255 time=0.407 ms
64 bytes from 10.23.40.211: icmp_seq=3 ttl=255 time=0.535 ms (DUP!)
64 bytes from 10.23.40.211: icmp_seq=4 ttl=255 time=0.406 ms
64 bytes from 10.23.40.211: icmp_seq=4 ttl=255 time=0.427 ms (DUP!)
64 bytes from 10.23.40.211: icmp_seq=5 ttl=255 time=0.400 ms
64 bytes from 10.23.40.211: icmp_seq=5 ttl=255 time=0.421 ms (DUP!)
64 bytes from 10.23.40.211: icmp_seq=6 ttl=255 time=0.402 ms
64 bytes from 10.23.40.211: icmp_seq=6 ttl=255 time=0.424 ms (DUP!)
64 bytes from 10.23.40.211: icmp_seq=7 ttl=255 time=0.528 ms
64 bytes from 10.23.40.211: icmp_seq=7 ttl=255 time=0.550 ms (DUP!)
64 bytes from 10.23.40.211: icmp_seq=8 ttl=255 time=0.516 ms
64 bytes from 10.23.40.211: icmp_seq=8 ttl=255 time=0.536 ms (DUP!)
64 bytes from 10.23.40.211: icmp_seq=9 ttl=255 time=0.515 ms
--- 10.23.40.211 ping statistics ---
10 packets transmitted, 10 packets received, +9 duplicates, 0.0% packet loss
round-trip min/avg/max/stddev = 0.391/0.477/0.633/0.069 ms
[root@boron ~]# snoop -d i40e0 'icmp'
Using device i40e0 (promiscuous mode)
iris.vlan40.acheron.be -> boron ICMP Echo request (ID: 7992 Sequence number: 0)
iris.vlan40.acheron.be -> boron ICMP Echo request (ID: 7992 Sequence number: 0)
boron -> iris.vlan40.acheron.be ICMP Echo reply (ID: 7992 Sequence number: 0)
boron -> iris.vlan40.acheron.be ICMP Echo reply (ID: 7992 Sequence number: 0)
iris.vlan40.acheron.be -> boron ICMP Echo request (ID: 7992 Sequence number: 1)
boron -> iris.vlan40.acheron.be ICMP Echo reply (ID: 7992 Sequence number: 1)
iris.vlan40.acheron.be -> boron ICMP Echo request (ID: 7992 Sequence number: 1)
boron -> iris.vlan40.acheron.be ICMP Echo reply (ID: 7992 Sequence number: 1)
iris.vlan40.acheron.be -> boron ICMP Echo request (ID: 7992 Sequence number: 2)
boron -> iris.vlan40.acheron.be ICMP Echo reply (ID: 7992 Sequence number: 2)
iris.vlan40.acheron.be -> boron ICMP Echo request (ID: 7992 Sequence number: 2)
boron -> iris.vlan40.acheron.be ICMP Echo reply (ID: 7992 Sequence number: 2)
iris.vlan40.acheron.be -> boron ICMP Echo request (ID: 7992 Sequence number: 3)
boron -> iris.vlan40.acheron.be ICMP Echo reply (ID: 7992 Sequence number: 3)
iris.vlan40.acheron.be -> boron ICMP Echo request (ID: 7992 Sequence number: 3)
boron -> iris.vlan40.acheron.be ICMP Echo reply (ID: 7992 Sequence number: 3)
iris.vlan40.acheron.be -> boron ICMP Echo request (ID: 7992 Sequence number: 4)
iris.vlan40.acheron.be -> boron ICMP Echo request (ID: 7992 Sequence number: 4)
boron -> iris.vlan40.acheron.be ICMP Echo reply (ID: 7992 Sequence number: 4)
boron -> iris.vlan40.acheron.be ICMP Echo reply (ID: 7992 Sequence number: 4)
iris.vlan40.acheron.be -> boron ICMP Echo request (ID: 7992 Sequence number: 5)
iris.vlan40.acheron.be -> boron ICMP Echo request (ID: 7992 Sequence number: 5)
boron -> iris.vlan40.acheron.be ICMP Echo reply (ID: 7992 Sequence number: 5)
boron -> iris.vlan40.acheron.be ICMP Echo reply (ID: 7992 Sequence number: 5)
iris.vlan40.acheron.be -> boron ICMP Echo request (ID: 7992 Sequence number: 6)
iris.vlan40.acheron.be -> boron ICMP Echo request (ID: 7992 Sequence number: 6)
boron -> iris.vlan40.acheron.be ICMP Echo reply (ID: 7992 Sequence number: 6)
boron -> iris.vlan40.acheron.be ICMP Echo reply (ID: 7992 Sequence number: 6)
iris.vlan40.acheron.be -> boron ICMP Echo request (ID: 7992 Sequence number: 7)
iris.vlan40.acheron.be -> boron ICMP Echo request (ID: 7992 Sequence number: 7)
boron -> iris.vlan40.acheron.be ICMP Echo reply (ID: 7992 Sequence number: 7)
boron -> iris.vlan40.acheron.be ICMP Echo reply (ID: 7992 Sequence number: 7)
iris.vlan40.acheron.be -> boron ICMP Echo request (ID: 7992 Sequence number: 8)
iris.vlan40.acheron.be -> boron ICMP Echo request (ID: 7992 Sequence number: 8)
boron -> iris.vlan40.acheron.be ICMP Echo reply (ID: 7992 Sequence number: 8)
boron -> iris.vlan40.acheron.be ICMP Echo reply (ID: 7992 Sequence number: 8)
iris.vlan40.acheron.be -> boron ICMP Echo request (ID: 7992 Sequence number: 9)
iris.vlan40.acheron.be -> boron ICMP Echo request (ID: 7992 Sequence number: 9)
boron -> iris.vlan40.acheron.be ICMP Echo reply (ID: 7992 Sequence number: 9)
boron -> iris.vlan40.acheron.be ICMP Echo reply (ID: 7992 Sequence number: 9)
Updated by 
Updated by 
Updated by