Project

General

Profile

Bug #13230

i40e has duplicate traffic when used with bhyve/snoop running

Added by Jorge Schrauwen about 2 months ago. Updated about 1 month ago.

Status:
New
Priority:
Normal
Assignee:
-
Category:
-
Start date:
Due date:
% Done:

0%

Estimated time:
Difficulty:
Medium
Tags:
Gerrit CR:

Description

After having my board replaced my 2x 10G copper i40e nics are now working... sort of.

I noticed I was getting duplicate traffic inside bhyve which broke IPv6 and PPPoE, also duplicate traffic is not good regardless of what it breaks.

Initial tests:
- turn of offloading in the VM (had issues with this in the past): no change
- switch to a different underlying nic, igb in this case: works fine without dups
- switch back to i40e: dups are back...
- tried a 1G link instead of a 10G link: dups remained.

Left the system to run over igb (1G) instead of i40e (10G) until I had more time to debug.

Second set of tests:
- eliminated bhyve and add a vnic with a vlan on top of i40e: no dups (oh shit was it bhyve? ha no it wasn't)
- run snoop on the vnic: dups return (aha! so I guess whatever viona/bhyve is doing snoop also does)
( I think someone else was seeing a lot of re-transmits too with snoop running a few weeks ago, maybe related?)
- eliminated vnic with vlan tagging: same result! no dups unless snoop is running, then the dups are back.

Setup information:

[root@boron ~]# dladm show-phys
LINK         MEDIA                STATE      SPEED  DUPLEX    DEVICE
i40e0        Ethernet             up         1000   full      i40e0
igb0         Ethernet             up         1000   full      igb0
i40e1        Ethernet             up         1000   full      i40e1

i40e0: admin nictag (no vlan tagging)
igb0: trunk nictag (vlan tagging)
i40e1: what I want to use as trunk nictag, now just there for testing.

[root@boron ~]# dladm show-vnic
LINK         OVER       SPEED MACADDRESS        MACADDRTYPE VID  ZONE
net0         igb0       0     82:fc:c:c2:e4:6f  fixed       110  eceec5eb-22c7-e65b-fd6b-81ebdb36406b
net1         igb0       0     0:22:6:a:0:1      fixed       110  99e40ee7-a8f9-4b57-9225-e7bd19f64b07
net2         igb0       0     0:22:6:14:0:1     fixed       120  99e40ee7-a8f9-4b57-9225-e7bd19f64b07
net3         igb0       0     0:22:6:1e:0:1     fixed       130  99e40ee7-a8f9-4b57-9225-e7bd19f64b07
net4         igb0       0     0:22:6:28:0:1     fixed       140  99e40ee7-a8f9-4b57-9225-e7bd19f64b07
net0         igb0       0     0:22:6:5:0:a      fixed       10   99e40ee7-a8f9-4b57-9225-e7bd19f64b07
vnic110      i40e1      1000  2:8:20:43:c8:a0   random      110  --

vnic110, vnic on top of i40e1 tagged with vlan 110

[root@boron ~]# ipadm show-addr
ADDROBJ           TYPE     STATE        ADDR
lo0/v4            static   ok           127.0.0.1/8
i40e0/_a          static   ok           10.23.40.211/24
vnic110/dhcp      dhcp     ok           10.23.10.225/24
lo0/v6            static   ok           ::1/128
i40e0/?           static   ok           fe80::3eec:efff:fe0c:783a/10
i40e0/_b          static   ok           2a02:578:470f:40::211/64
i40e0/?           addrconf ok           2a02:578:470f:40:3eec:efff:fe0c:783a/64
i40e0/?           dhcp     ok           2a02:578:470f:40::246/128

10.23.40.211 gz-ip without vlan tagging
10.23.10.225 gz-ip with vlan tagging

Ping results without snoop running:

[iris :: sjorge][~]
[.]$ ping -c 10 10.23.40.211
PING 10.23.40.211 (10.23.40.211): 56 data bytes
64 bytes from 10.23.40.211: icmp_seq=0 ttl=255 time=0.745 ms
64 bytes from 10.23.40.211: icmp_seq=1 ttl=255 time=0.248 ms
64 bytes from 10.23.40.211: icmp_seq=2 ttl=255 time=0.234 ms
64 bytes from 10.23.40.211: icmp_seq=3 ttl=255 time=0.238 ms
64 bytes from 10.23.40.211: icmp_seq=4 ttl=255 time=0.349 ms
64 bytes from 10.23.40.211: icmp_seq=5 ttl=255 time=0.223 ms
64 bytes from 10.23.40.211: icmp_seq=6 ttl=255 time=0.224 ms
64 bytes from 10.23.40.211: icmp_seq=7 ttl=255 time=0.345 ms
64 bytes from 10.23.40.211: icmp_seq=8 ttl=255 time=0.339 ms
64 bytes from 10.23.40.211: icmp_seq=9 ttl=255 time=0.333 ms

--- 10.23.40.211 ping statistics ---
10 packets transmitted, 10 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 0.223/0.328/0.745/0.148 ms

[hyperon :: sjorge][~]
[■]$ ping -c 10 10.23.10.225
PING 10.23.10.225 (10.23.10.225): 56 data bytes
64 bytes from 10.23.10.225: icmp_seq=0 ttl=255 time=0.429 ms
64 bytes from 10.23.10.225: icmp_seq=1 ttl=255 time=0.420 ms
64 bytes from 10.23.10.225: icmp_seq=2 ttl=255 time=0.702 ms
64 bytes from 10.23.10.225: icmp_seq=3 ttl=255 time=0.523 ms
64 bytes from 10.23.10.225: icmp_seq=4 ttl=255 time=0.593 ms
64 bytes from 10.23.10.225: icmp_seq=5 ttl=255 time=0.593 ms
64 bytes from 10.23.10.225: icmp_seq=6 ttl=255 time=0.605 ms
64 bytes from 10.23.10.225: icmp_seq=7 ttl=255 time=0.575 ms
64 bytes from 10.23.10.225: icmp_seq=8 ttl=255 time=0.396 ms
64 bytes from 10.23.10.225: icmp_seq=9 ttl=255 time=0.683 ms

--- 10.23.10.225 ping statistics ---
10 packets transmitted, 10 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 0.396/0.552/0.702/0.102 ms

Pings were done from macOS for 10.23.10.0/24 and from a freebsd host for 10.23.40.0/24... to make sure there was no router in play.

Ping results with snoop running:

[hyperon :: sjorge][~]
[■]$ ping -c 10 10.23.10.225
PING 10.23.10.225 (10.23.10.225): 56 data bytes
64 bytes from 10.23.10.225: icmp_seq=0 ttl=255 time=0.575 ms
64 bytes from 10.23.10.225: icmp_seq=0 ttl=255 time=0.580 ms (DUP!)
64 bytes from 10.23.10.225: icmp_seq=1 ttl=255 time=8.708 ms
64 bytes from 10.23.10.225: icmp_seq=1 ttl=255 time=8.711 ms (DUP!)
64 bytes from 10.23.10.225: icmp_seq=2 ttl=255 time=0.447 ms
64 bytes from 10.23.10.225: icmp_seq=2 ttl=255 time=0.452 ms (DUP!)
64 bytes from 10.23.10.225: icmp_seq=3 ttl=255 time=0.446 ms
64 bytes from 10.23.10.225: icmp_seq=3 ttl=255 time=0.451 ms (DUP!)
64 bytes from 10.23.10.225: icmp_seq=4 ttl=255 time=0.389 ms
64 bytes from 10.23.10.225: icmp_seq=4 ttl=255 time=0.436 ms (DUP!)
64 bytes from 10.23.10.225: icmp_seq=5 ttl=255 time=0.589 ms
64 bytes from 10.23.10.225: icmp_seq=5 ttl=255 time=0.597 ms (DUP!)
64 bytes from 10.23.10.225: icmp_seq=6 ttl=255 time=0.601 ms
64 bytes from 10.23.10.225: icmp_seq=6 ttl=255 time=0.613 ms (DUP!)
64 bytes from 10.23.10.225: icmp_seq=7 ttl=255 time=0.398 ms
64 bytes from 10.23.10.225: icmp_seq=7 ttl=255 time=0.611 ms (DUP!)
64 bytes from 10.23.10.225: icmp_seq=8 ttl=255 time=0.586 ms
64 bytes from 10.23.10.225: icmp_seq=8 ttl=255 time=0.622 ms (DUP!)
64 bytes from 10.23.10.225: icmp_seq=9 ttl=255 time=8.514 ms

--- 10.23.10.225 ping statistics ---
10 packets transmitted, 10 packets received, +9 duplicates, 0.0% packet loss
round-trip min/avg/max/stddev = 0.389/1.807/8.711/2.962 ms

[root@boron ~]# snoop -d vnic110 'icmp'
Using device vnic110 (promiscuous mode)
hyperon.vlan10.acheron.be -> 10.23.10.225 ICMP Echo request (ID: 50479 Sequence number: 0)
hyperon.vlan10.acheron.be -> 10.23.10.225 ICMP Echo request (ID: 50479 Sequence number: 0)
10.23.10.225 -> hyperon.vlan10.acheron.be ICMP Echo reply (ID: 50479 Sequence number: 0)
10.23.10.225 -> hyperon.vlan10.acheron.be ICMP Echo reply (ID: 50479 Sequence number: 0)
hyperon.vlan10.acheron.be -> 10.23.10.225 ICMP Echo request (ID: 50479 Sequence number: 1)
hyperon.vlan10.acheron.be -> 10.23.10.225 ICMP Echo request (ID: 50479 Sequence number: 1)
10.23.10.225 -> hyperon.vlan10.acheron.be ICMP Echo reply (ID: 50479 Sequence number: 1)
10.23.10.225 -> hyperon.vlan10.acheron.be ICMP Echo reply (ID: 50479 Sequence number: 1)
hyperon.vlan10.acheron.be -> 10.23.10.225 ICMP Echo request (ID: 50479 Sequence number: 2)
10.23.10.225 -> hyperon.vlan10.acheron.be ICMP Echo reply (ID: 50479 Sequence number: 2)
hyperon.vlan10.acheron.be -> 10.23.10.225 ICMP Echo request (ID: 50479 Sequence number: 2)
10.23.10.225 -> hyperon.vlan10.acheron.be ICMP Echo reply (ID: 50479 Sequence number: 2)
hyperon.vlan10.acheron.be -> 10.23.10.225 ICMP Echo request (ID: 50479 Sequence number: 3)
hyperon.vlan10.acheron.be -> 10.23.10.225 ICMP Echo request (ID: 50479 Sequence number: 3)
10.23.10.225 -> hyperon.vlan10.acheron.be ICMP Echo reply (ID: 50479 Sequence number: 3)
10.23.10.225 -> hyperon.vlan10.acheron.be ICMP Echo reply (ID: 50479 Sequence number: 3)
hyperon.vlan10.acheron.be -> 10.23.10.225 ICMP Echo request (ID: 50479 Sequence number: 4)
hyperon.vlan10.acheron.be -> 10.23.10.225 ICMP Echo request (ID: 50479 Sequence number: 4)
10.23.10.225 -> hyperon.vlan10.acheron.be ICMP Echo reply (ID: 50479 Sequence number: 4)
10.23.10.225 -> hyperon.vlan10.acheron.be ICMP Echo reply (ID: 50479 Sequence number: 4)
hyperon.vlan10.acheron.be -> 10.23.10.225 ICMP Echo request (ID: 50479 Sequence number: 5)
hyperon.vlan10.acheron.be -> 10.23.10.225 ICMP Echo request (ID: 50479 Sequence number: 5)
10.23.10.225 -> hyperon.vlan10.acheron.be ICMP Echo reply (ID: 50479 Sequence number: 5)
10.23.10.225 -> hyperon.vlan10.acheron.be ICMP Echo reply (ID: 50479 Sequence number: 5)
hyperon.vlan10.acheron.be -> 10.23.10.225 ICMP Echo request (ID: 50479 Sequence number: 6)
hyperon.vlan10.acheron.be -> 10.23.10.225 ICMP Echo request (ID: 50479 Sequence number: 6)
10.23.10.225 -> hyperon.vlan10.acheron.be ICMP Echo reply (ID: 50479 Sequence number: 6)
10.23.10.225 -> hyperon.vlan10.acheron.be ICMP Echo reply (ID: 50479 Sequence number: 6)
hyperon.vlan10.acheron.be -> 10.23.10.225 ICMP Echo request (ID: 50479 Sequence number: 7)
hyperon.vlan10.acheron.be -> 10.23.10.225 ICMP Echo request (ID: 50479 Sequence number: 7)
10.23.10.225 -> hyperon.vlan10.acheron.be ICMP Echo reply (ID: 50479 Sequence number: 7)
10.23.10.225 -> hyperon.vlan10.acheron.be ICMP Echo reply (ID: 50479 Sequence number: 7)
hyperon.vlan10.acheron.be -> 10.23.10.225 ICMP Echo request (ID: 50479 Sequence number: 8)
10.23.10.225 -> hyperon.vlan10.acheron.be ICMP Echo reply (ID: 50479 Sequence number: 8)
hyperon.vlan10.acheron.be -> 10.23.10.225 ICMP Echo request (ID: 50479 Sequence number: 8)
10.23.10.225 -> hyperon.vlan10.acheron.be ICMP Echo reply (ID: 50479 Sequence number: 8)
hyperon.vlan10.acheron.be -> 10.23.10.225 ICMP Echo request (ID: 50479 Sequence number: 9)
hyperon.vlan10.acheron.be -> 10.23.10.225 ICMP Echo request (ID: 50479 Sequence number: 9)
10.23.10.225 -> hyperon.vlan10.acheron.be ICMP Echo reply (ID: 50479 Sequence number: 9)
10.23.10.225 -> hyperon.vlan10.acheron.be ICMP Echo reply (ID: 50479 Sequence number: 9)

[iris :: sjorge][~]
[.]$ ping -c 10 10.23.40.211
PING 10.23.40.211 (10.23.40.211): 56 data bytes
64 bytes from 10.23.40.211: icmp_seq=0 ttl=255 time=0.524 ms
64 bytes from 10.23.40.211: icmp_seq=0 ttl=255 time=0.633 ms (DUP!)
64 bytes from 10.23.40.211: icmp_seq=1 ttl=255 time=0.402 ms
64 bytes from 10.23.40.211: icmp_seq=1 ttl=255 time=0.519 ms (DUP!)
64 bytes from 10.23.40.211: icmp_seq=2 ttl=255 time=0.391 ms
64 bytes from 10.23.40.211: icmp_seq=2 ttl=255 time=0.518 ms (DUP!)
64 bytes from 10.23.40.211: icmp_seq=3 ttl=255 time=0.407 ms
64 bytes from 10.23.40.211: icmp_seq=3 ttl=255 time=0.535 ms (DUP!)
64 bytes from 10.23.40.211: icmp_seq=4 ttl=255 time=0.406 ms
64 bytes from 10.23.40.211: icmp_seq=4 ttl=255 time=0.427 ms (DUP!)
64 bytes from 10.23.40.211: icmp_seq=5 ttl=255 time=0.400 ms
64 bytes from 10.23.40.211: icmp_seq=5 ttl=255 time=0.421 ms (DUP!)
64 bytes from 10.23.40.211: icmp_seq=6 ttl=255 time=0.402 ms
64 bytes from 10.23.40.211: icmp_seq=6 ttl=255 time=0.424 ms (DUP!)
64 bytes from 10.23.40.211: icmp_seq=7 ttl=255 time=0.528 ms
64 bytes from 10.23.40.211: icmp_seq=7 ttl=255 time=0.550 ms (DUP!)
64 bytes from 10.23.40.211: icmp_seq=8 ttl=255 time=0.516 ms
64 bytes from 10.23.40.211: icmp_seq=8 ttl=255 time=0.536 ms (DUP!)
64 bytes from 10.23.40.211: icmp_seq=9 ttl=255 time=0.515 ms

--- 10.23.40.211 ping statistics ---
10 packets transmitted, 10 packets received, +9 duplicates, 0.0% packet loss
round-trip min/avg/max/stddev = 0.391/0.477/0.633/0.069 ms

[root@boron ~]# snoop -d i40e0 'icmp'
Using device i40e0 (promiscuous mode)
iris.vlan40.acheron.be -> boron        ICMP Echo request (ID: 7992 Sequence number: 0)
iris.vlan40.acheron.be -> boron        ICMP Echo request (ID: 7992 Sequence number: 0)
       boron -> iris.vlan40.acheron.be ICMP Echo reply (ID: 7992 Sequence number: 0)
       boron -> iris.vlan40.acheron.be ICMP Echo reply (ID: 7992 Sequence number: 0)
iris.vlan40.acheron.be -> boron        ICMP Echo request (ID: 7992 Sequence number: 1)
       boron -> iris.vlan40.acheron.be ICMP Echo reply (ID: 7992 Sequence number: 1)
iris.vlan40.acheron.be -> boron        ICMP Echo request (ID: 7992 Sequence number: 1)
       boron -> iris.vlan40.acheron.be ICMP Echo reply (ID: 7992 Sequence number: 1)
iris.vlan40.acheron.be -> boron        ICMP Echo request (ID: 7992 Sequence number: 2)
       boron -> iris.vlan40.acheron.be ICMP Echo reply (ID: 7992 Sequence number: 2)
iris.vlan40.acheron.be -> boron        ICMP Echo request (ID: 7992 Sequence number: 2)
       boron -> iris.vlan40.acheron.be ICMP Echo reply (ID: 7992 Sequence number: 2)
iris.vlan40.acheron.be -> boron        ICMP Echo request (ID: 7992 Sequence number: 3)
       boron -> iris.vlan40.acheron.be ICMP Echo reply (ID: 7992 Sequence number: 3)
iris.vlan40.acheron.be -> boron        ICMP Echo request (ID: 7992 Sequence number: 3)
       boron -> iris.vlan40.acheron.be ICMP Echo reply (ID: 7992 Sequence number: 3)
iris.vlan40.acheron.be -> boron        ICMP Echo request (ID: 7992 Sequence number: 4)
iris.vlan40.acheron.be -> boron        ICMP Echo request (ID: 7992 Sequence number: 4)
       boron -> iris.vlan40.acheron.be ICMP Echo reply (ID: 7992 Sequence number: 4)
       boron -> iris.vlan40.acheron.be ICMP Echo reply (ID: 7992 Sequence number: 4)
iris.vlan40.acheron.be -> boron        ICMP Echo request (ID: 7992 Sequence number: 5)
iris.vlan40.acheron.be -> boron        ICMP Echo request (ID: 7992 Sequence number: 5)
       boron -> iris.vlan40.acheron.be ICMP Echo reply (ID: 7992 Sequence number: 5)
       boron -> iris.vlan40.acheron.be ICMP Echo reply (ID: 7992 Sequence number: 5)
iris.vlan40.acheron.be -> boron        ICMP Echo request (ID: 7992 Sequence number: 6)
iris.vlan40.acheron.be -> boron        ICMP Echo request (ID: 7992 Sequence number: 6)
       boron -> iris.vlan40.acheron.be ICMP Echo reply (ID: 7992 Sequence number: 6)
       boron -> iris.vlan40.acheron.be ICMP Echo reply (ID: 7992 Sequence number: 6)
iris.vlan40.acheron.be -> boron        ICMP Echo request (ID: 7992 Sequence number: 7)
iris.vlan40.acheron.be -> boron        ICMP Echo request (ID: 7992 Sequence number: 7)
       boron -> iris.vlan40.acheron.be ICMP Echo reply (ID: 7992 Sequence number: 7)
       boron -> iris.vlan40.acheron.be ICMP Echo reply (ID: 7992 Sequence number: 7)
iris.vlan40.acheron.be -> boron        ICMP Echo request (ID: 7992 Sequence number: 8)
iris.vlan40.acheron.be -> boron        ICMP Echo request (ID: 7992 Sequence number: 8)
       boron -> iris.vlan40.acheron.be ICMP Echo reply (ID: 7992 Sequence number: 8)
       boron -> iris.vlan40.acheron.be ICMP Echo reply (ID: 7992 Sequence number: 8)
iris.vlan40.acheron.be -> boron        ICMP Echo request (ID: 7992 Sequence number: 9)
iris.vlan40.acheron.be -> boron        ICMP Echo request (ID: 7992 Sequence number: 9)
       boron -> iris.vlan40.acheron.be ICMP Echo reply (ID: 7992 Sequence number: 9)
       boron -> iris.vlan40.acheron.be ICMP Echo reply (ID: 7992 Sequence number: 9)
#1

Updated by Jorge Schrauwen about 2 months ago

The nics are build into a https://www.supermicro.com/en/products/motherboard/X11SPH-nCTF

So it's Intel X722 + X557 PHY

#2

Updated by Jorge Schrauwen about 2 months ago

[17:47:28] rzezeski: sjorge: I couldn't reproduce, try running this DTrace script while doing your pings with dup traffic: https://gist.github.com/rzezeski/6e99738bddc7f3a5fd0b90f9c6c4d187
[17:47:44] rzezeski: run with `dtrace -Cs /path/to/file.d`
RESERVED    1/13   b8:27:eb:5f:a7:4b 3c:ec:ef:0c:78:3a 0800 0    10.23.40.42     10.23.40.211    1   
DEFAULT     0/1    b8:27:eb:5f:a7:4b 3c:ec:ef:0c:78:3a 0800 0    10.23.40.42     10.23.40.211    1   
RESERVED    1/13   b8:27:eb:5f:a7:4b 3c:ec:ef:0c:78:3a 0800 0    10.23.40.42     10.23.40.211    1   
DEFAULT     0/1    b8:27:eb:5f:a7:4b 3c:ec:ef:0c:78:3a 0800 0    10.23.40.42     10.23.40.211    1   
RESERVED    1/13   b8:27:eb:5f:a7:4b 3c:ec:ef:0c:78:3a 0800 0    10.23.40.42     10.23.40.211    1   
DEFAULT     0/1    b8:27:eb:5f:a7:4b 3c:ec:ef:0c:78:3a 0800 0    10.23.40.42     10.23.40.211    1   
RESERVED    1/13   b8:27:eb:5f:a7:4b 3c:ec:ef:0c:78:3a 0800 0    10.23.40.42     10.23.40.211    1   
DEFAULT     0/1    b8:27:eb:5f:a7:4b 3c:ec:ef:0c:78:3a 0800 0    10.23.40.42     10.23.40.211    1   
RESERVED    1/13   b8:27:eb:5f:a7:4b 3c:ec:ef:0c:78:3a 0800 0    10.23.40.42     10.23.40.211    1   
DEFAULT     0/1    b8:27:eb:5f:a7:4b 3c:ec:ef:0c:78:3a 0800 0    10.23.40.42     10.23.40.211    1   
RESERVED    1/13   b8:27:eb:5f:a7:4b 3c:ec:ef:0c:78:3a 0800 0    10.23.40.42     10.23.40.211    1   
DEFAULT     0/1    b8:27:eb:5f:a7:4b 3c:ec:ef:0c:78:3a 0800 0    10.23.40.42     10.23.40.211    1   
RESERVED    1/13   b8:27:eb:5f:a7:4b 3c:ec:ef:0c:78:3a 0800 0    10.23.40.42     10.23.40.211    1   
DEFAULT     0/1    b8:27:eb:5f:a7:4b 3c:ec:ef:0c:78:3a 0800 0    10.23.40.42     10.23.40.211    1   
RESERVED    1/13   b8:27:eb:5f:a7:4b 3c:ec:ef:0c:78:3a 0800 0    10.23.40.42     10.23.40.211    1   
DEFAULT     0/1    b8:27:eb:5f:a7:4b 3c:ec:ef:0c:78:3a 0800 0    10.23.40.42     10.23.40.211    1   
RESERVED    1/13   b8:27:eb:5f:a7:4b 3c:ec:ef:0c:78:3a 0800 0    10.23.40.42     10.23.40.211    1   
DEFAULT     0/1    b8:27:eb:5f:a7:4b 3c:ec:ef:0c:78:3a 0800 0    10.23.40.42     10.23.40.211    1   
RESERVED    1/13   b8:27:eb:5f:a7:4b 3c:ec:ef:0c:78:3a 0800 0    10.23.40.42     10.23.40.211    1   
DEFAULT     0/1    b8:27:eb:5f:a7:4b 3c:ec:ef:0c:78:3a 0800 0    10.23.40.42     10.23.40.211    1   
RESERVED    1/13   b8:27:eb:5f:a7:4b 3c:ec:ef:0c:78:3a 0800 0    10.23.40.42     10.23.40.211    1   
DEFAULT     0/1    b8:27:eb:5f:a7:4b 3c:ec:ef:0c:78:3a 0800 0    10.23.40.42     10.23.40.211    1   
RESERVED    1/13   b8:27:eb:5f:a7:4b 3c:ec:ef:0c:78:3a 0800 0    10.23.40.42     10.23.40.211    1   
DEFAULT     0/1    b8:27:eb:5f:a7:4b 3c:ec:ef:0c:78:3a 0800 0    10.23.40.42     10.23.40.211    1   

RESERVED    1/13   b8:27:eb:5f:a7:4b 3c:ec:ef:0c:78:3a 0800 0    10.23.40.42     10.23.40.211    1   
DEFAULT     0/1    b8:27:eb:5f:a7:4b 3c:ec:ef:0c:78:3a 0800 0    10.23.40.42     10.23.40.211    1   
#3

Updated by Jorge Schrauwen about 2 months ago


RESERVED    2/17   00:22:06:03:10:01 02:08:20:43:c8:a0 0800 110  10.23.10.150    10.23.10.225    1   
DEFAULT     0/0    00:22:06:03:10:01 02:08:20:43:c8:a0 0800 110  10.23.10.150    10.23.10.225    1   
RESERVED    2/17   00:22:06:03:10:01 02:08:20:43:c8:a0 0800 110  10.23.10.150    10.23.10.225    1   
DEFAULT     0/0    00:22:06:03:10:01 02:08:20:43:c8:a0 0800 110  10.23.10.150    10.23.10.225    1   
RESERVED    2/17   00:22:06:03:10:01 02:08:20:43:c8:a0 0800 110  10.23.10.150    10.23.10.225    1   
DEFAULT     0/0    00:22:06:03:10:01 02:08:20:43:c8:a0 0800 110  10.23.10.150    10.23.10.225    1   
RESERVED    2/17   00:22:06:03:10:01 02:08:20:43:c8:a0 0800 110  10.23.10.150    10.23.10.225    1   
DEFAULT     0/0    00:22:06:03:10:01 02:08:20:43:c8:a0 0800 110  10.23.10.150    10.23.10.225    1   
RESERVED    2/17   DEFAULT     0/0    00:22:06:03:10:01 02:08:20:43:c8:a0 0800 110  10.23.10.150    10.23.10.225    1   
00:22:06:03:10:01 02:08:20:43:c8:a0 0800 110  10.23.10.150    10.23.10.225    1   
RESERVED    2/17   00:22:06:03:10:01 02:08:20:43:c8:a0 0800 110  10.23.10.150    10.23.10.225    1   
DEFAULT     0/0    00:22:06:03:10:01 02:08:20:43:c8:a0 0800 110  10.23.10.150    10.23.10.225    1   
RESERVED    2/17   00:22:06:03:10:01 02:08:20:43:c8:a0 0800 110  10.23.10.150    10.23.10.225    1   
DEFAULT     0/0    00:22:06:03:10:01 02:08:20:43:c8:a0 0800 110  10.23.10.150    10.23.10.225    1   
RESERVED    2/17   00:22:06:03:10:01 02:08:20:43:c8:a0 0800 110  10.23.10.150    10.23.10.225    1   
DEFAULT     0/0    00:22:06:03:10:01 02:08:20:43:c8:a0 0800 110  10.23.10.150    10.23.10.225    1   
RESERVED    2/17   00:22:06:03:10:01 02:08:20:43:c8:a0 0800 110  10.23.10.150    10.23.10.225    1   
DEFAULT     0/0    00:22:06:03:10:01 02:08:20:43:c8:a0 0800 110  10.23.10.150    10.23.10.225    1   
RESERVED    2/17   00:22:06:03:10:01 02:08:20:43:c8:a0 0800 110  10.23.10.150    10.23.10.225    1   
DEFAULT     0/0    00:22:06:03:10:01 02:08:20:43:c8:a0 0800 110  10.23.10.150    10.23.10.225    1   
RESERVED    2/17   00:22:06:03:10:01 02:08:20:43:c8:a0 0800 110  10.23.10.150    10.23.10.225    1   
DEFAULT     0/0    00:22:06:03:10:01 02:08:20:43:c8:a0 0800 110  10.23.10.150    10.23.10.225    1   
RESERVED    2/17   00:22:06:03:10:01 02:08:20:43:c8:a0 0800 110  10.23.10.150    10.23.10.225    1   
DEFAULT     0/0    00:22:06:03:10:01 02:08:20:43:c8:a0 0800 110  10.23.10.150    10.23.10.225    1   
RESERVED    2/17   00:22:06:03:10:01 02:08:20:43:c8:a0 0800 110  10.23.10.150    10.23.10.225    1   
DEFAULT     0/0    00:22:06:03:10:01 02:08:20:43:c8:a0 0800 110  10.23.10.150    10.23.10.225    1   
RESERVED    2/17   00:22:06:03:10:01 02:08:20:43:c8:a0 0800 110  10.23.10.150    10.23.10.225    1   
DEFAULT     0/0    00:22:06:03:10:01 02:08:20:43:c8:a0 0800 110  10.23.10.150    10.23.10.225    1   
RESERVED    2/17   00:22:06:03:10:01 02:08:20:43:c8:a0 0800 110  10.23.10.150    10.23.10.225    1   
DEFAULT     0/0    00:22:06:03:10:01 02:08:20:43:c8:a0 0800 110  10.23.10.150    10.23.10.225    1   
RESERVED    2/17   00:22:06:03:10:01 02:08:20:43:c8:a0 0800 110  10.23.10.150    10.23.10.225    1   
DEFAULT     0/0    00:22:06:03:10:01 02:08:20:43:c8:a0 0800 110  10.23.10.150    10.23.10.225    1   
RESERVED    2/17   00:22:06:03:10:01 02:08:20:43:c8:a0 0800 110  10.23.10.150    10.23.10.225    1   
DEFAULT     0/0    00:22:06:03:10:01 02:08:20:43:c8:a0 0800 110  10.23.10.150    10.23.10.225    1   
RESERVED    2/17   00:22:06:03:10:01 02:08:20:43:c8:a0 0800 110  10.23.10.150    10.23.10.225    1   
DEFAULT     0/0    00:22:06:03:10:01 02:08:20:43:c8:a0 0800 110  10.23.10.150    10.23.10.225    1   
RESERVED    2/17   00:22:06:03:10:01 02:08:20:43:c8:a0 0800 110  10.23.10.150    10.23.10.225    1   
DEFAULT     0/0    00:22:06:03:10:01 02:08:20:43:c8:a0 0800 110  10.23.10.150    10.23.10.225    1   
RESERVED    2/17   00:22:06:03:10:01 02:08:20:43:c8:a0 0800 110  10.23.10.150    10.23.10.225    1   
DEFAULT     0/0    00:22:06:03:10:01 02:08:20:43:c8:a0 0800 110  10.23.10.150    10.23.10.225    1   
RESERVED    2/17   00:22:06:03:10:01 02:08:20:43:c8:a0 0800 110  10.23.10.150    10.23.10.225    1   
RESERVED    2/17   00:22:06:03:10:01 02:08:20:43:c8:a0 0800 110  10.23.10.150    10.23.10.225    1   
RESERVED    2/17   00:22:06:03:10:01 02:08:20:43:c8:a0 0800 110  10.23.10.150    10.23.10.225    1   
RESERVED    2/17   00:22:06:03:10:01 02:08:20:43:c8:a0 0800 110  10.23.10.150    10.23.10.225    1   
RESERVED    2/17   00:22:06:03:10:01 02:08:20:43:c8:a0 0800 110  10.23.10.150    10.23.10.225    1   
RESERVED    2/17   00:22:06:03:10:01 02:08:20:43:c8:a0 0800 110  10.23.10.150    10.23.10.225    1   

This one was with the vnic on top, I killed the snoop need the end and the default one goes away

#4

Updated by Ryan Zezeski about 2 months ago

The DTrace output indicates that the unicast traffic is hitting both the reserved group (as it should) and the default group (which it should not). We program the mac providers so that unicast traffic flows to its reserved group (if one is available) and multicast/broadcast/all other traffic flow to the default group so it can be software classified. When promisc mode is enabled the reserved group unicast traffic should remain untouched, but all other traffic on the L2 network should arrive on the default group. This allows all traffic to reach mac_rx_common() and thus be delivered to mac_promisc_dispatch(). In this case, since the unicast traffic is being delivered to both the reserved group and default group, we are having duplicate calls to mac_rx_common(). And since this traffic is destined for an existing client it means we see duplicate traffic both in snoop AND at the client itself (once for direct delivery via the hardware ring/SRS, and once for the default ring/mac_rx_classify()).

I do not encounter the same issue on my i40e part (X710-DA2). This indicates either some type of L2 steering change in the firmware or perhaps a logic bug in how we program these parts that works by dumb luck in some cases. More research is needed into the hardware/firmware differences and perhaps a peek at the latest PRM.

#5

Updated by Jorge Schrauwen about 2 months ago

Firmware is the latest available from SuperMicro for this board/nic combo.

Labeled as NVM v4.11, not sure in the PXE/iSCSI blobs, but probably the NVM is the interesting one

fw 3.1.55727 api 1.5 nvm 4.11 etid 80001da4 oem 1.265.0

#6

Updated by Jorge Schrauwen about 1 month ago

root@fermion:~ # tcpdump -s0 -i ixl0.110 icmp
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on ixl0.110, link-type EN10MB (Ethernet), capture size 262144 bytes
08:38:39.390648 IP tachyon.vlan10.acheron.be > fermion.acheron.be: ICMP echo request, id 25026, seq 61, length 64
08:38:39.390911 IP fermion.acheron.be > tachyon.vlan10.acheron.be: ICMP echo reply, id 25026, seq 61, length 64
08:38:40.393513 IP tachyon.vlan10.acheron.be > fermion.acheron.be: ICMP echo request, id 25026, seq 62, length 64
08:38:40.393790 IP fermion.acheron.be > tachyon.vlan10.acheron.be: ICMP echo reply, id 25026, seq 62, length 64
08:38:41.395851 IP tachyon.vlan10.acheron.be > fermion.acheron.be: ICMP echo request, id 25026, seq 63, length 64
08:38:41.396146 IP fermion.acheron.be > tachyon.vlan10.acheron.be: ICMP echo reply, id 25026, seq 63, length 64
08:38:42.398967 IP tachyon.vlan10.acheron.be > fermion.acheron.be: ICMP echo request, id 25026, seq 64, length 64
08:38:42.399234 IP fermion.acheron.be > tachyon.vlan10.acheron.be: ICMP echo reply, id 25026, seq 64, length 64
08:38:43.401270 IP tachyon.vlan10.acheron.be > fermion.acheron.be: ICMP echo request, id 25026, seq 65, length 64
08:38:43.401512 IP fermion.acheron.be > tachyon.vlan10.acheron.be: ICMP echo reply, id 25026, seq 65, length 64
08:38:44.405983 IP tachyon.vlan10.acheron.be > fermion.acheron.be: ICMP echo request, id 25026, seq 66, length 64
08:38:44.406258 IP fermion.acheron.be > tachyon.vlan10.acheron.be: ICMP echo reply, id 25026, seq 66, length 64
08:38:45.411345 IP tachyon.vlan10.acheron.be > fermion.acheron.be: ICMP echo request, id 25026, seq 67, length 64
08:38:45.411634 IP fermion.acheron.be > tachyon.vlan10.acheron.be: ICMP echo reply, id 25026, seq 67, length 64
^C
14 packets captured
24 packets received by filter
0 packets dropped by kernel

Same NIC passthru to a freebsd VM via bhyve, works fine there when running the interface in promisc mode.

Seems likely we're programming it wrong somewhere.

Edit: this was with the latest intel provided driver for freebsd (https://downloadcenter.intel.com/download/25160/Intel-Network-Adapter-Driver-for-Intel-Ethernet-Controller-700-Series-under-FreeBSD-?wapkw=x722)

#7

Updated by Jorge Schrauwen about 1 month ago

Device[104] /devices/pci@5e,0/pci8086,2030@0/pci8086,37c0@0/pci8086,37c5@3/pci15d9,37d2@0
Class [ufm]
        Vendor: Intel Corporation
        Device: Ethernet Connection X722 for 10GBASE-T
        Capabilities: Report
        Image 0: Firmware
            Slot 0 (-|-|a): 4.1

prtconf -Dv

                    pci15d9,37d2, instance #0 (driver name: i40e)
                        Driver properties:
                            name='api-version' type=string items=1 dev=none
                                value='1.9'
                            name='firmware-build' type=string items=1 dev=none
                                value='e70c'
                            name='firmware-version' type=string items=1 dev=none
                                value='4.1'
                            name='printed-board-assembly' type=string items=1 dev=none
                                value=''
                            name='fm-accchk-capable' type=boolean dev=none
                            name='fm-dmachk-capable' type=boolean dev=none
                            name='fm-errcb-capable' type=boolean dev=none
                            name='fm-ereport-capable' type=boolean dev=none
                        Hardware properties:
                            name='ddi-ufm-capable' type=boolean
                            name='pci-msix-capid-pointer' type=int items=1
                                value=00000070
                            name='pci-msi-capid-pointer' type=int items=1
                                value=00000050
                            name='pcie-link-supported-speeds' type=int64 items=1
                                value=000000009502f900
                            name='pcie-link-maximum-speed' type=int64 items=1
                                value=000000009502f900
                            name='pcie-link-current-speed' type=int64 items=1
                                value=000000009502f900
                            name='pcie-link-current-width' type=int items=1
                                value=00000001
                            name='pcie-link-maximum-width' type=int items=1
                                value=00000001
                            name='pcie-aspm-state' type=string items=1
                                value='disabled'
                            name='pcie-aspm-support' type=string items=1
                                value='l0s,l1'
                            name='pcie-serial' type=byte items=8
                                value=3c.ec.ef.ff.ff.0c.78.3a
                            name='acpi-namespace' type=string items=1
                                value='\_SB_.PC02.BR2A.EPCU.VSP3.FPK0'
                            name='assigned-addresses' type=int items=10
                                value=c3670010.00000000.df000000.00000000.01000000.c367001c.00000000.e0008000.00000000.00008000
                            name='reg' type=int items=15
                                value=00670000.00000000.00000000.00000000.00000000.43670010.00000000.00000000.00000000.01000000.4367001c.00000000.00000000.00000000.00008000
                            name='compatible' type=string items=15
                                value='pciex8086,37d2.15d9.37d2.9' + 'pciex8086,37d2.15d9.37d2' + 'pciex8086,37d2.9' + 'pciex8086,37d2' + 'pciexclass,020000' + 'pciexclass,0200' + 'pci8086,37d2.15d9.37d2.9' + 'pci8086,37d2.15d9.37d2' + 'pci15d9,37d2,s' + 'pci15d9,37d2' + 'pci8086,37d2.9' + 'pci8086,37d2,p' + 'pci8086,37d2' + 'pciclass,020000' + 'pciclass,0200'
                            name='model' type=string items=1
                                value='Ethernet controller'
                            name='power-consumption' type=int items=2
                                value=00000001.00000001
                            name='devsel-speed' type=int items=1
                                value=00000000
                            name='interrupts' type=int items=1
                                value=00000001
                            name='subsystem-vendor-id' type=int items=1
                                value=000015d9
                            name='subsystem-id' type=int items=1
                                value=000037d2
                            name='unit-address' type=string items=1
                                value='0'
                            name='class-code' type=int items=1
                                value=00020000
                            name='revision-id' type=int items=1
                                value=00000009
                            name='vendor-id' type=int items=1
                                value=00008086
                            name='device-id' type=int items=1
                                value=000037d2
                            name='vendor-name' type=string items=1
                                value='Intel Corporation'
                            name='device-name' type=string items=1
                                value='Ethernet Connection X722 for 10GBASE-T'
                            name='subsystem-name' type=string items=1
                                value='unknown subsystem'
                        Device Minor Nodes:
                            dev=(296,1)
                                dev_path=/pci@5e,0/pci8086,2030@0/pci8086,37c0@0/pci8086,37c5@3/pci15d9,37d2@0:i40e0
                                    spectype=chr type=minor
                                    dev_link=/dev/i40e0
                            dev=(296,1006)
                                dev_path=<clone>
                                Device Minor Layered Under:
                                    mod=udp6 accesstype=chr
                                        dev_path=/pseudo/udp6@0
                            dev=(296,1004)
                                dev_path=<clone>
                                Device Minor Layered Under:
                                    mod=udp accesstype=chr
                                        dev_path=/pseudo/udp@0
                            dev=(296,1003)
                                dev_path=<clone>
                                Device Minor Layered Under:
                                    mod=udp accesstype=chr
                                        dev_path=/pseudo/udp@0

Also available in: Atom PDF