Project

General

Profile

Actions
Copy link

Bug #12011

closed

ixgbe reports incorrect MAC_STAT_NORCVBUF

Added by Ryan Zezeski over 3 years ago. Updated over 3 years ago.

Status:
Closed
Priority:
Normal
Assignee:
Ryan Zezeski
Category:
driver - device drivers
Start date:
Due date:
% Done:

100%

Estimated time:
Difficulty:
Medium
Tags:
Gerrit CR:
External Bug:

Description

The MAC_STAT_NORCVBUF will always report zero on all controllers other than 82598. The RNBC register it relies on was removed starting with 82599. From then on you query the queue dropped packet via the QPRDC[n] registers. Our current driver (incoherently) maps the RQMSR[n].Q_MAP[m] registers that the QPDRC[n] registers rely on, but it never reads the later. We should map the RQMSR reg in a way that makes more sense and use the QPDRC reg to fill in MAC_STAT_NORCVBUF for non-82598 controllers.

Actions
Copy link
 #1

Updated by Ryan Zezeski over 3 years ago

  • Status changed from New to In Progress
Actions
Copy link
 #2

Updated by Ryan Zezeski over 3 years ago

While working on some feedback from Robert I noticed a few additional issues. I'm going to lump them in with this ticket unless someone objects.

First, I'm removing the various queue statistic arrays for both Rx and Tx. E.g., instead of having 16 qbrc stats (Queue Bytes Received Counter) we will now have one qor (Queue Octets Received). While doing this work I found a few supplemental issues.

  • The comment about the GORC/GOTC regs being unreliable is now over 12 years old. Let's assume Intel fixed the glitch and use the correct registers (instead of using the tor/tot values, which are also wrong).
  • The gprc/gptc (Good Packets) values were being filled by the qprc/qptc values -- this is just wrong. There are separate good packet regs provided by the controller, use them.
  • The tor (Total Octets Received) was being filled by the qbrc -- also wrong. The controller provides the IXGBE_TOR{H,L} regs for this.
  • The tot (Total Octets Transmitted) is a nonexistent stat on the controller but we use it to populate MAC_STAT_OBYTES. We fill it from the queue regs, but a better solution is to fill it from the got value. It makes sense that the only octets transmitted are the good octets (though it makes you wonder why they have a TPT reg).
Actions
Copy link
 #3

Updated by Ryan Zezeski over 3 years ago

I built this as both DEBUG and non-DEBUG, with gcc4 shadow, and smatch on.

I tested this on an 82599ES part and an X540 part.

To test I ran an iperf3 benchmark using a 64 byte buffer which makes it easy to overwhelm the Rx path on the server and cause queue drops (norcvbuf kstat). I captured the relevant kstats before and after the run, making sure that total packets/octets looked reasonable and that good_pkts_recvd - queue_pkts_recvd == norcvbuf.

82599ES data

                    name='device-name' type=string items=1
                        value='82599ES 10-Gigabit SFI/SFP+ Network Connection'
                    name='subsystem-name' type=string items=1
                        value='Ethernet 10Gb 2-port 560SFP+ Adapter'

[root@e4-11-5b-97-83-48 (emy-15) ~]# kstat -p ixgbe:::good_* ixgbe:::queue_* ixgbe::mac:norcvbuf ixgbe::mac:*bytes64
ixgbe:0:statistics:good_octets_recvd    24560
ixgbe:0:statistics:good_octets_xmitd    8420
ixgbe:0:statistics:good_pkts_recvd      80
ixgbe:0:statistics:good_pkts_xmitd      79
ixgbe:1:statistics:good_octets_recvd    841939
ixgbe:1:statistics:good_octets_xmitd    887855
ixgbe:1:statistics:good_pkts_recvd      7783
ixgbe:1:statistics:good_pkts_xmitd      8028
ixgbe:0:statistics:queue_octets_recvd   24240
ixgbe:0:statistics:queue_octets_xmitd   8104
ixgbe:0:statistics:queue_pkts_recvd     80
ixgbe:0:statistics:queue_pkts_xmitd     79
ixgbe:1:statistics:queue_octets_recvd   810597
ixgbe:1:statistics:queue_octets_xmitd   851249
ixgbe:1:statistics:queue_pkts_recvd     7780
ixgbe:1:statistics:queue_pkts_xmitd     8028
ixgbe:0:mac:norcvbuf    0
ixgbe:1:mac:norcvbuf    3
ixgbe:0:mac:obytes64    8420
ixgbe:0:mac:rbytes64    248726
ixgbe:1:mac:obytes64    887855
ixgbe:1:mac:rbytes64    847929

[root@e4-11-5b-97-83-48 (emy-15) ~]# LD_PRELOAD=/var/tmp/rpz/iperf3/libiperf.so.0 /var/tmp/rpz/iperf3/iperf3 -s -B 172.25.15.81 -fm -l 64
-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------
Accepted connection from 172.25.15.4, port 46455
[  5] local 172.25.15.81 port 5201 connected to 172.25.15.4 port 56803
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec  5.87 MBytes  49.3 Mbits/sec                  
[  5]   1.00-2.00   sec  2.85 MBytes  23.9 Mbits/sec                  
[  5]   2.00-3.00   sec  7.05 MBytes  59.2 Mbits/sec                  
[  5]   3.00-4.00   sec  3.29 MBytes  27.6 Mbits/sec                  
[  5]   4.00-5.00   sec  2.67 MBytes  22.4 Mbits/sec                  
[  5]   5.00-6.00   sec  2.48 MBytes  20.8 Mbits/sec                  
[  5]   6.00-7.00   sec  2.14 MBytes  17.9 Mbits/sec                  
[  5]   7.00-8.00   sec  3.97 MBytes  33.3 Mbits/sec                  
[  5]   8.00-9.00   sec  2.01 MBytes  16.8 Mbits/sec                  
[  5]   9.00-10.00  sec  6.09 MBytes  51.0 Mbits/sec                  
[  5]  10.00-11.00  sec  3.29 MBytes  27.6 Mbits/sec                  
[  5]  11.00-12.00  sec  3.84 MBytes  32.2 Mbits/sec                  
[  5]  12.00-13.00  sec  14.3 MBytes   120 Mbits/sec                  
[  5]  13.00-14.00  sec  14.4 MBytes   121 Mbits/sec                  
[  5]  14.00-15.00  sec  14.5 MBytes   122 Mbits/sec                  
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-15.00  sec  88.7 MBytes  49.6 Mbits/sec                  receiver
-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------
^Ciperf3: interrupt - the server has terminated

[root@e4-11-5b-97-83-48 (emy-15) ~]# kstat -p ixgbe:::good_* ixgbe:::queue_* ixgbe::mac:norcvbuf ixgbe::mac:*bytes64
ixgbe:0:statistics:good_octets_recvd    25788
ixgbe:0:statistics:good_octets_xmitd    8848
ixgbe:0:statistics:good_pkts_recvd      84
ixgbe:0:statistics:good_pkts_xmitd      83
ixgbe:1:statistics:good_octets_recvd    196602409
ixgbe:1:statistics:good_octets_xmitd    59492421
ixgbe:1:statistics:good_pkts_recvd      1302225
ixgbe:1:statistics:good_pkts_xmitd      749791
ixgbe:0:statistics:queue_octets_recvd   25452
ixgbe:0:statistics:queue_octets_xmitd   8516
ixgbe:0:statistics:queue_pkts_recvd     84
ixgbe:0:statistics:queue_pkts_xmitd     83
ixgbe:1:statistics:queue_octets_recvd   179181979
ixgbe:1:statistics:queue_octets_xmitd   56488565
ixgbe:1:statistics:queue_pkts_recvd     1212758
ixgbe:1:statistics:queue_pkts_xmitd     749791
ixgbe:0:mac:norcvbuf    0
ixgbe:1:mac:norcvbuf    89467
ixgbe:0:mac:obytes64    8848
ixgbe:0:mac:rbytes64    263906
ixgbe:1:mac:obytes64    59492421
ixgbe:1:mac:rbytes64    196608719

Confirmed the norcvbuf:

$ bc <<<"1302225 - 1212758" 
89467

X540 data

                    name='device-name' type=string items=1
                        value='Ethernet Controller 10-Gigabit X540-AT2'
                    name='subsystem-name' type=string items=1
                        value='Ethernet Converged Network Adapter X540-T2'

<GZ> root@sys76 [~]
# kstat -p ixgbe:::good_* ixgbe:::queue_* ixgbe::mac:norcvbuf ixgbe::mac:*bytes64
ixgbe:0:statistics:good_octets_recvd    49009
ixgbe:0:statistics:good_octets_xmitd    49009
ixgbe:0:statistics:good_pkts_recvd      552
ixgbe:0:statistics:good_pkts_xmitd      552
ixgbe:1:statistics:good_octets_recvd    48935
ixgbe:1:statistics:good_octets_xmitd    49009
ixgbe:1:statistics:good_pkts_recvd      551
ixgbe:1:statistics:good_pkts_xmitd      552
ixgbe:0:statistics:queue_octets_recvd   46801
ixgbe:0:statistics:queue_octets_xmitd   45041
ixgbe:0:statistics:queue_pkts_recvd     552
ixgbe:0:statistics:queue_pkts_xmitd     552
ixgbe:1:statistics:queue_octets_recvd   46731
ixgbe:1:statistics:queue_octets_xmitd   45041
ixgbe:1:statistics:queue_pkts_recvd     551
ixgbe:1:statistics:queue_pkts_xmitd     552
ixgbe:0:mac:norcvbuf    0
ixgbe:1:mac:norcvbuf    0
ixgbe:0:mac:obytes64    49009
ixgbe:0:mac:rbytes64    49009
ixgbe:1:mac:obytes64    49009
ixgbe:1:mac:rbytes64    49009

iperf3 run

[root@wall-market ~]# iperf3 -c 192.168.3.103 -B 192.168.3.102 -l 64 -N -t 15
Connecting to host 192.168.3.103, port 5201
[  4] local 192.168.3.102 port 41975 connected to 192.168.3.103 port 5201
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-1.00   sec  17.7 MBytes   149 Mbits/sec                  
[  4]   1.00-2.00   sec  27.2 MBytes   229 Mbits/sec                  
[  4]   2.00-3.00   sec  22.0 MBytes   185 Mbits/sec                  
[  4]   3.00-4.00   sec  18.4 MBytes   154 Mbits/sec                  
[  4]   4.00-5.00   sec  23.1 MBytes   193 Mbits/sec                  
[  4]   5.00-6.00   sec  25.9 MBytes   218 Mbits/sec                  
[  4]   6.00-7.00   sec  22.9 MBytes   192 Mbits/sec                  
[  4]   7.00-8.00   sec  25.4 MBytes   213 Mbits/sec                  
[  4]   8.00-9.00   sec  20.7 MBytes   174 Mbits/sec                  
[  4]   9.00-10.00  sec  20.8 MBytes   174 Mbits/sec                  
[  4]  10.00-11.00  sec  20.3 MBytes   170 Mbits/sec                  
[  4]  11.00-12.00  sec  24.8 MBytes   208 Mbits/sec                  
[  4]  12.00-13.00  sec  25.5 MBytes   214 Mbits/sec                  
[  4]  13.00-14.00  sec  22.1 MBytes   185 Mbits/sec                  
[  4]  14.00-15.00  sec  25.8 MBytes   217 Mbits/sec                  
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-15.00  sec   343 MBytes   192 Mbits/sec                  sender
[  4]   0.00-15.00  sec   336 MBytes   188 Mbits/sec                  receiver

iperf Done.

<GZ> root@sys76 [~]
# kstat -p ixgbe:::good_* ixgbe:::queue_* ixgbe::mac:norcvbuf ixgbe::mac:*bytes64
ixgbe:0:statistics:good_octets_recvd    100322748
ixgbe:0:statistics:good_octets_xmitd    559103688
ixgbe:0:statistics:good_pkts_recvd      1419353
ixgbe:0:statistics:good_pkts_xmitd      2887673
ixgbe:1:statistics:good_octets_recvd    559103614
ixgbe:1:statistics:good_octets_xmitd    100322748
ixgbe:1:statistics:good_pkts_recvd      2887672
ixgbe:1:statistics:good_pkts_xmitd      1419353
ixgbe:0:statistics:queue_octets_recvd   83267408
ixgbe:0:statistics:queue_octets_xmitd   547551218
ixgbe:0:statistics:queue_pkts_recvd     1247681
ixgbe:0:statistics:queue_pkts_xmitd     2887673
ixgbe:1:statistics:queue_octets_recvd   544818040
ixgbe:1:statistics:queue_octets_xmitd   94643552
ixgbe:1:statistics:queue_pkts_recvd     2866857
ixgbe:1:statistics:queue_pkts_xmitd     1419353
ixgbe:0:mac:norcvbuf    171672
ixgbe:1:mac:norcvbuf    20815
ixgbe:0:mac:obytes64    559103688
ixgbe:0:mac:rbytes64    100322748
ixgbe:1:mac:obytes64    100322748
ixgbe:1:mac:rbytes64    559103688

If we take the delta between good Rx packets and queued Rx packets we
should end up with norcvbuf.

ixgbe0

$ bc <<<"1419353 - 1247681" 
171672

ixgbe1

$ bc <<<"2887672 - 2866857" 
20815

Actions
Copy link
 #4

Updated by Electric Monk over 3 years ago

  • Status changed from In Progress to Closed
  • % Done changed from 0 to 100

git commit 168e1ed4b6c6d87d390ab3eb29cc9e9cb948ef56

commit  168e1ed4b6c6d87d390ab3eb29cc9e9cb948ef56
Author: Ryan Zezeski <rpz@joyent.com>
Date:   2019-12-11T19:48:54.000Z

    12011 ixgbe reports incorrect MAC_STAT_NORCVBUF
    Reviewed by: Robert Mustacchi <rm@fingolfin.org>
    Reviewed by: Dan McDonald <danmcd@joyent.com>
    Approved by: Gordon Ross <gordon.w.ross@gmail.com>

Actions
Copy link

Also available in: Atom PDF