Project

General

Profile

Bug #11546

Track TCP round-trip time in nanoseconds

Added by Cody Mello 7 days ago. Updated about 1 hour ago.

Status:
Closed
Priority:
Normal
Assignee:
-
Category:
networking
Start date:
Due date:
% Done:

100%

Estimated time:
Difficulty:
Medium
Tags:

Description

As part of upstreaming Delphix's connstat(1M) changes, we'll want to pull in their updates to TCP to use nanosecond timestamps for RTT instead of microseconds. This includes changes made in DLPX-42721 and DLPX-43064. This change will also make TCP stop using echoed timestamps for calculating RTT; when committed, we can also close out #8725.

This work was initially done by Sebastien Roy and Brandon Baker at Delphix, with some further changes made by Cody Mello at Joyent in OS-7334.

History

#1

Updated by Cody Mello 3 days ago

These changes allow for measuring a more precise RTT, but that shouldn't have any impact on the performance. To verify this, I ran some iperf tests between two systems with 40GiB Chelsio NICs. On a recent SmartOS PI that doesn't include these changes (joyent_20190328T010321Z):

[root@ivy ~]# /var/tmp/iperf3-static-sunos -c 10.200.200.202 -p 11845 -t 120
Connecting to host 10.200.200.202, port 11845
[  5] local 10.200.200.201 port 63240 connected to 10.200.200.202 port 11845
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec   425 MBytes  3.56 Gbits/sec
[  5]   1.00-2.00   sec   693 MBytes  5.81 Gbits/sec
[  5]   2.00-3.00   sec   450 MBytes  3.78 Gbits/sec
[  5]   3.00-4.00   sec   658 MBytes  5.52 Gbits/sec
[  5]   4.00-5.00   sec   455 MBytes  3.82 Gbits/sec
[  5]   5.00-6.00   sec   450 MBytes  3.78 Gbits/sec
[  5]   6.00-7.00   sec  1.91 GBytes  16.4 Gbits/sec
[  5]   7.00-8.00   sec   452 MBytes  3.79 Gbits/sec
[  5]   8.00-9.00   sec   478 MBytes  4.01 Gbits/sec
[  5]   9.00-10.00  sec   450 MBytes  3.77 Gbits/sec
[  5]  10.00-11.00  sec  1.17 GBytes  10.0 Gbits/sec
[  5]  11.00-12.00  sec   635 MBytes  5.33 Gbits/sec
[  5]  12.00-13.00  sec   572 MBytes  4.79 Gbits/sec
<snip>
[  5] 100.00-101.00 sec  1.94 GBytes  16.6 Gbits/sec
[  5] 101.00-102.00 sec  1.93 GBytes  16.5 Gbits/sec
[  5] 102.00-103.00 sec  1.94 GBytes  16.6 Gbits/sec
[  5] 103.00-104.00 sec  1.94 GBytes  16.6 Gbits/sec
[  5] 104.00-105.00 sec  1.92 GBytes  16.5 Gbits/sec
[  5] 105.00-106.00 sec  1.93 GBytes  16.6 Gbits/sec
[  5] 106.00-107.00 sec  1.93 GBytes  16.6 Gbits/sec
[  5] 107.00-108.00 sec  1.92 GBytes  16.5 Gbits/sec
[  5] 108.00-109.00 sec  1.93 GBytes  16.6 Gbits/sec
[  5] 109.00-110.00 sec  1.92 GBytes  16.5 Gbits/sec
[  5] 110.00-111.00 sec  1.92 GBytes  16.5 Gbits/sec
[  5] 111.00-112.00 sec  1.92 GBytes  16.5 Gbits/sec
[  5] 112.00-113.00 sec  1.92 GBytes  16.5 Gbits/sec
[  5] 113.00-114.00 sec  1.93 GBytes  16.6 Gbits/sec
[  5] 114.00-115.00 sec  1.92 GBytes  16.5 Gbits/sec
[  5] 115.00-116.00 sec  1.92 GBytes  16.5 Gbits/sec
[  5] 116.00-117.00 sec  1.91 GBytes  16.4 Gbits/sec
[  5] 117.00-118.00 sec  1.91 GBytes  16.4 Gbits/sec
[  5] 118.00-119.00 sec  1.92 GBytes  16.5 Gbits/sec
[  5] 119.00-120.00 sec  1.93 GBytes  16.6 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-120.00 sec   157 GBytes  11.2 Gbits/sec                  sender
[  5]   0.00-120.00 sec   157 GBytes  11.2 Gbits/sec                  receiver

iperf Done.
[root@ivy ~]# /var/tmp/iperf3-static-sunos -c 10.200.200.202 -p 11845 -t 120
Connecting to host 10.200.200.202, port 11845
[  5] local 10.200.200.201 port 53375 connected to 10.200.200.202 port 11845
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec   421 MBytes  3.53 Gbits/sec
[  5]   1.00-2.00   sec  1.17 GBytes  10.1 Gbits/sec
[  5]   2.00-3.00   sec   844 MBytes  7.08 Gbits/sec
[  5]   3.00-4.00   sec   516 MBytes  4.33 Gbits/sec
[  5]   4.00-5.00   sec   653 MBytes  5.47 Gbits/sec
[  5]   5.00-6.00   sec   712 MBytes  5.97 Gbits/sec
[  5]   6.00-7.00   sec  1.92 GBytes  16.5 Gbits/sec
[  5]   7.00-8.00   sec  1.96 GBytes  16.8 Gbits/sec
[  5]   8.00-9.00   sec  1.96 GBytes  16.8 Gbits/sec
[  5]   9.00-10.00  sec  1.95 GBytes  16.8 Gbits/sec
[  5]  10.00-11.00  sec  1.97 GBytes  16.9 Gbits/sec
[  5]  11.00-12.00  sec  1.97 GBytes  17.0 Gbits/sec
[  5]  12.00-13.00  sec  1.98 GBytes  17.0 Gbits/sec
[  5]  13.00-14.00  sec  1.98 GBytes  17.0 Gbits/sec
<snip>
[  5] 100.00-101.00 sec  1.97 GBytes  16.9 Gbits/sec
[  5] 101.00-102.00 sec  1.97 GBytes  16.9 Gbits/sec
[  5] 102.00-103.00 sec  1.97 GBytes  16.9 Gbits/sec
[  5] 103.00-104.00 sec  1.97 GBytes  16.9 Gbits/sec
[  5] 104.00-105.00 sec  1.97 GBytes  16.9 Gbits/sec
[  5] 105.00-106.00 sec  1.97 GBytes  17.0 Gbits/sec
[  5] 106.00-107.00 sec  1.97 GBytes  16.9 Gbits/sec
[  5] 107.00-108.00 sec  1.95 GBytes  16.8 Gbits/sec
[  5] 108.00-109.00 sec  1.96 GBytes  16.9 Gbits/sec
[  5] 109.00-110.00 sec  1.96 GBytes  16.9 Gbits/sec
[  5] 110.00-111.00 sec  1.97 GBytes  17.0 Gbits/sec
[  5] 111.00-112.00 sec  1.98 GBytes  17.0 Gbits/sec
[  5] 112.00-113.00 sec  1.96 GBytes  16.9 Gbits/sec
[  5] 113.00-114.00 sec  1.95 GBytes  16.8 Gbits/sec
[  5] 114.00-115.00 sec  1.95 GBytes  16.8 Gbits/sec
[  5] 115.00-116.00 sec  1.95 GBytes  16.8 Gbits/sec
[  5] 116.00-117.00 sec  1.95 GBytes  16.8 Gbits/sec
[  5] 117.00-118.00 sec  1.95 GBytes  16.8 Gbits/sec
[  5] 118.00-119.00 sec  1.95 GBytes  16.8 Gbits/sec
[  5] 119.00-120.00 sec  1.96 GBytes  16.8 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-120.00 sec   228 GBytes  16.3 Gbits/sec                  sender
[  5]   0.00-120.00 sec   228 GBytes  16.3 Gbits/sec                  receiver

iperf Done.
[root@ivy ~]# /var/tmp/iperf3-static-sunos -c 10.200.200.202 -p 11845 -t 120 -P 4
Connecting to host 10.200.200.202, port 11845
[  5] local 10.200.200.201 port 64322 connected to 10.200.200.202 port 11845
[  7] local 10.200.200.201 port 35299 connected to 10.200.200.202 port 11845
[  9] local 10.200.200.201 port 51000 connected to 10.200.200.202 port 11845
[ 11] local 10.200.200.201 port 36272 connected to 10.200.200.202 port 11845
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec   637 MBytes  5.34 Gbits/sec
[  7]   0.00-1.00   sec   835 MBytes  7.01 Gbits/sec
[  9]   0.00-1.00   sec   417 MBytes  3.50 Gbits/sec
[ 11]   0.00-1.00   sec   630 MBytes  5.28 Gbits/sec
[SUM]   0.00-1.00   sec  2.46 GBytes  21.1 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   1.00-2.00   sec   659 MBytes  5.53 Gbits/sec
[  7]   1.00-2.00   sec   734 MBytes  6.15 Gbits/sec
[  9]   1.00-2.00   sec   655 MBytes  5.50 Gbits/sec
[ 11]   1.00-2.00   sec   612 MBytes  5.13 Gbits/sec
[SUM]   1.00-2.00   sec  2.60 GBytes  22.3 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   2.00-3.00   sec   626 MBytes  5.25 Gbits/sec
[  7]   2.00-3.00   sec   693 MBytes  5.81 Gbits/sec
[  9]   2.00-3.00   sec   620 MBytes  5.20 Gbits/sec
[ 11]   2.00-3.00   sec   864 MBytes  7.25 Gbits/sec
[SUM]   2.00-3.00   sec  2.74 GBytes  23.5 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
<snip>
[  5] 118.00-119.00 sec   616 MBytes  5.17 Gbits/sec
[  7] 118.00-119.00 sec   655 MBytes  5.49 Gbits/sec
[  9] 118.00-119.00 sec   602 MBytes  5.05 Gbits/sec
[ 11] 118.00-119.00 sec   736 MBytes  6.17 Gbits/sec
[SUM] 118.00-119.00 sec  2.55 GBytes  21.9 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5] 119.00-120.00 sec   614 MBytes  5.16 Gbits/sec
[  7] 119.00-120.00 sec   688 MBytes  5.78 Gbits/sec
[  9] 119.00-120.00 sec   599 MBytes  5.03 Gbits/sec
[ 11] 119.00-120.00 sec   764 MBytes  6.42 Gbits/sec
[SUM] 119.00-120.00 sec  2.60 GBytes  22.4 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-120.00 sec  71.0 GBytes  5.08 Gbits/sec                  sender
[  5]   0.00-120.00 sec  71.0 GBytes  5.08 Gbits/sec                  receiver
[  7]   0.00-120.00 sec  80.6 GBytes  5.77 Gbits/sec                  sender
[  7]   0.00-120.00 sec  80.6 GBytes  5.77 Gbits/sec                  receiver
[  9]   0.00-120.00 sec  70.4 GBytes  5.04 Gbits/sec                  sender
[  9]   0.00-120.00 sec  70.4 GBytes  5.04 Gbits/sec                  receiver
[ 11]   0.00-120.00 sec  89.2 GBytes  6.38 Gbits/sec                  sender
[ 11]   0.00-120.00 sec  89.2 GBytes  6.38 Gbits/sec                  receiver
[SUM]   0.00-120.00 sec   311 GBytes  22.3 Gbits/sec                  sender
[SUM]   0.00-120.00 sec   311 GBytes  22.3 Gbits/sec                  receiver

iperf Done.
[root@ivy ~]# /var/tmp/iperf3-static-sunos -c 10.200.200.202 -p 11845 -t 120 -P 4
Connecting to host 10.200.200.202, port 11845
[  5] local 10.200.200.201 port 41045 connected to 10.200.200.202 port 11845
[  7] local 10.200.200.201 port 51532 connected to 10.200.200.202 port 11845
[  9] local 10.200.200.201 port 55819 connected to 10.200.200.202 port 11845
[ 11] local 10.200.200.201 port 49547 connected to 10.200.200.202 port 11845
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec   740 MBytes  6.21 Gbits/sec                  
[  7]   0.00-1.00   sec   582 MBytes  4.88 Gbits/sec                  
[  9]   0.00-1.00   sec   554 MBytes  4.65 Gbits/sec                  
[ 11]   0.00-1.00   sec   408 MBytes  3.42 Gbits/sec                  
[SUM]   0.00-1.00   sec  2.23 GBytes  19.2 Gbits/sec                  
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   1.00-2.00   sec   721 MBytes  6.05 Gbits/sec                  
[  7]   1.00-2.00   sec   584 MBytes  4.90 Gbits/sec                  
[  9]   1.00-2.00   sec   724 MBytes  6.07 Gbits/sec                  
[ 11]   1.00-2.00   sec   490 MBytes  4.11 Gbits/sec                  
[SUM]   1.00-2.00   sec  2.46 GBytes  21.1 Gbits/sec                  
- - - - - - - - - - - - - - - - - - - - - - - - -
<snip>
[  5] 118.00-119.00 sec   525 MBytes  4.41 Gbits/sec                  
[  7] 118.00-119.00 sec   900 MBytes  7.55 Gbits/sec                  
[  9] 118.00-119.00 sec   459 MBytes  3.85 Gbits/sec                  
[ 11] 118.00-119.00 sec   818 MBytes  6.86 Gbits/sec                  
[SUM] 118.00-119.00 sec  2.64 GBytes  22.7 Gbits/sec                  
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5] 119.00-120.00 sec   502 MBytes  4.21 Gbits/sec                  
[  7] 119.00-120.00 sec   915 MBytes  7.68 Gbits/sec                  
[  9] 119.00-120.00 sec   430 MBytes  3.61 Gbits/sec                  
[ 11] 119.00-120.00 sec   863 MBytes  7.24 Gbits/sec                  
[SUM] 119.00-120.00 sec  2.65 GBytes  22.7 Gbits/sec                  
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-120.00 sec  63.6 GBytes  4.56 Gbits/sec                  sender
[  5]   0.00-120.00 sec  63.6 GBytes  4.56 Gbits/sec                  receiver
[  7]   0.00-120.00 sec  98.2 GBytes  7.03 Gbits/sec                  sender
[  7]   0.00-120.00 sec  98.2 GBytes  7.03 Gbits/sec                  receiver
[  9]   0.00-120.00 sec  54.8 GBytes  3.92 Gbits/sec                  sender
[  9]   0.00-120.00 sec  54.8 GBytes  3.92 Gbits/sec                  receiver
[ 11]   0.00-120.00 sec  93.8 GBytes  6.71 Gbits/sec                  sender
[ 11]   0.00-120.00 sec  93.8 GBytes  6.71 Gbits/sec                  receiver
[SUM]   0.00-120.00 sec   310 GBytes  22.2 Gbits/sec                  sender
[SUM]   0.00-120.00 sec   310 GBytes  22.2 Gbits/sec                  receiver

iperf Done.

On a SmartOS platform built with these changes (and #11547):

[root@ivy ~]# /var/tmp/iperf3-static-sunos -c 10.200.200.202 -p 11845 -t 120
Connecting to host 10.200.200.202, port 11845
[  5] local 10.200.200.201 port 45276 connected to 10.200.200.202 port 11845
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec   755 MBytes  6.34 Gbits/sec
[  5]   1.00-2.00   sec  1.09 GBytes  9.37 Gbits/sec
[  5]   2.00-3.00   sec   451 MBytes  3.78 Gbits/sec
[  5]   3.00-4.00   sec   451 MBytes  3.79 Gbits/sec
[  5]   4.00-5.00   sec   450 MBytes  3.77 Gbits/sec
[  5]   5.00-6.00   sec  1.16 GBytes  10.0 Gbits/sec
[  5]   6.00-7.00   sec   859 MBytes  7.20 Gbits/sec
[  5]   7.00-8.00   sec   619 MBytes  5.19 Gbits/sec
[  5]   8.00-9.00   sec   984 MBytes  8.26 Gbits/sec
[  5]   9.00-10.00  sec   735 MBytes  6.17 Gbits/sec
[  5]  10.00-11.00  sec   916 MBytes  7.68 Gbits/sec
[  5]  11.00-12.00  sec  1.59 GBytes  13.6 Gbits/sec
[  5]  12.00-13.00  sec   974 MBytes  8.17 Gbits/sec
[  5]  13.00-14.00  sec   759 MBytes  6.37 Gbits/sec
[  5]  14.00-15.00  sec  1.74 GBytes  14.9 Gbits/sec
[  5]  15.00-16.00  sec  1.16 GBytes  9.93 Gbits/sec
[  5]  16.00-17.00  sec   453 MBytes  3.80 Gbits/sec
[  5]  17.00-18.00  sec   952 MBytes  7.99 Gbits/sec
[  5]  18.00-19.00  sec   546 MBytes  4.58 Gbits/sec
[  5]  19.00-20.00  sec  1.18 GBytes  10.1 Gbits/sec
[  5]  20.00-21.00  sec  1.41 GBytes  12.1 Gbits/sec
[  5]  21.00-22.00  sec  1.98 GBytes  17.0 Gbits/sec
[  5]  22.00-23.00  sec  1.99 GBytes  17.1 Gbits/sec
[  5]  23.00-24.00  sec  1.98 GBytes  17.0 Gbits/sec
[  5]  24.00-25.00  sec  1.98 GBytes  17.0 Gbits/sec
[  5]  25.00-26.00  sec  1.99 GBytes  17.1 Gbits/sec
<snip>

[  5] 100.00-101.00 sec  1.92 GBytes  16.5 Gbits/sec
[  5] 101.00-102.00 sec  1.93 GBytes  16.6 Gbits/sec
[  5] 102.00-103.00 sec  1.92 GBytes  16.5 Gbits/sec
[  5] 103.00-104.00 sec  1.92 GBytes  16.5 Gbits/sec
[  5] 104.00-105.00 sec  1.93 GBytes  16.5 Gbits/sec
[  5] 105.00-106.00 sec  1.92 GBytes  16.5 Gbits/sec
[  5] 106.00-107.00 sec  1.92 GBytes  16.5 Gbits/sec
[  5] 107.00-108.00 sec  1.92 GBytes  16.5 Gbits/sec
[  5] 108.00-109.00 sec  1.92 GBytes  16.5 Gbits/sec
[  5] 109.00-110.00 sec  1.92 GBytes  16.5 Gbits/sec
[  5] 110.00-111.00 sec  1.94 GBytes  16.7 Gbits/sec
[  5] 111.00-112.00 sec  1.94 GBytes  16.7 Gbits/sec
[  5] 112.00-113.00 sec  1.94 GBytes  16.7 Gbits/sec
[  5] 113.00-114.00 sec  1.94 GBytes  16.7 Gbits/sec
[  5] 114.00-115.00 sec  1.94 GBytes  16.7 Gbits/sec
[  5] 115.00-116.00 sec  1.92 GBytes  16.5 Gbits/sec
[  5] 116.00-117.00 sec  1.92 GBytes  16.5 Gbits/sec
[  5] 117.00-118.00 sec  1.92 GBytes  16.5 Gbits/sec
[  5] 118.00-119.00 sec  1.92 GBytes  16.5 Gbits/sec
[  5] 119.00-120.00 sec  1.92 GBytes  16.5 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-120.00 sec   202 GBytes  14.5 Gbits/sec                  sender
[  5]   0.00-120.00 sec   202 GBytes  14.5 Gbits/sec                  receiver

iperf Done.
[root@ivy ~]# /var/tmp/iperf3-static-sunos -c 10.200.200.202 -p 11845 -t 120
Connecting to host 10.200.200.202, port 11845
[  5] local 10.200.200.201 port 33259 connected to 10.200.200.202 port 11845
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec   506 MBytes  4.25 Gbits/sec
[  5]   1.00-2.00   sec  1.14 GBytes  9.79 Gbits/sec
[  5]   2.00-3.00   sec   640 MBytes  5.37 Gbits/sec
[  5]   3.00-4.00   sec   450 MBytes  3.78 Gbits/sec
[  5]   4.00-5.00   sec   515 MBytes  4.32 Gbits/sec
[  5]   5.00-6.00   sec   689 MBytes  5.78 Gbits/sec
[  5]   6.00-7.00   sec   486 MBytes  4.07 Gbits/sec
[  5]   7.00-8.00   sec   659 MBytes  5.53 Gbits/sec
[  5]   8.00-9.00   sec   573 MBytes  4.81 Gbits/sec
[  5]   9.00-10.00  sec  1017 MBytes  8.53 Gbits/sec
[  5]  10.00-11.00  sec   453 MBytes  3.80 Gbits/sec
[  5]  11.00-12.00  sec   460 MBytes  3.86 Gbits/sec
[  5]  12.00-13.00  sec  1.58 GBytes  13.6 Gbits/sec
[  5]  13.00-14.00  sec  1.10 GBytes  9.46 Gbits/sec
[  5]  14.00-15.00  sec  1.97 GBytes  16.9 Gbits/sec
[  5]  15.00-16.00  sec  1.97 GBytes  16.9 Gbits/sec
[  5]  16.00-17.00  sec  1.96 GBytes  16.9 Gbits/sec
[  5]  17.00-18.00  sec  1.96 GBytes  16.9 Gbits/sec
[  5]  18.00-19.00  sec  1.97 GBytes  16.9 Gbits/sec
[  5]  19.00-20.00  sec  1.96 GBytes  16.9 Gbits/sec
<snip>
[  5] 100.00-101.00 sec  1.97 GBytes  16.9 Gbits/sec
[  5] 101.00-102.00 sec  1.98 GBytes  17.0 Gbits/sec
[  5] 102.00-103.00 sec  1.98 GBytes  17.0 Gbits/sec
[  5] 103.00-104.00 sec  1.98 GBytes  17.1 Gbits/sec
[  5] 104.00-105.00 sec  1.98 GBytes  17.0 Gbits/sec
[  5] 105.00-106.00 sec  1.99 GBytes  17.1 Gbits/sec
[  5] 106.00-107.00 sec  1.99 GBytes  17.1 Gbits/sec
[  5] 107.00-108.00 sec  1.96 GBytes  16.9 Gbits/sec
[  5] 108.00-109.00 sec  1.95 GBytes  16.8 Gbits/sec
[  5] 109.00-110.00 sec  1.96 GBytes  16.9 Gbits/sec
[  5] 110.00-111.00 sec  1.96 GBytes  16.8 Gbits/sec
[  5] 111.00-112.00 sec  1.95 GBytes  16.8 Gbits/sec
[  5] 112.00-113.00 sec  1.95 GBytes  16.7 Gbits/sec
[  5] 113.00-114.00 sec  1.96 GBytes  16.8 Gbits/sec
[  5] 114.00-115.00 sec  1.97 GBytes  16.9 Gbits/sec
[  5] 115.00-116.00 sec  1.97 GBytes  16.9 Gbits/sec
[  5] 116.00-117.00 sec  1.96 GBytes  16.8 Gbits/sec
[  5] 117.00-118.00 sec  1.96 GBytes  16.8 Gbits/sec
[  5] 118.00-119.00 sec  1.96 GBytes  16.8 Gbits/sec
[  5] 119.00-120.00 sec  1.96 GBytes  16.8 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-120.00 sec   218 GBytes  15.6 Gbits/sec                  sender
[  5]   0.00-120.00 sec   218 GBytes  15.6 Gbits/sec                  receiver

iperf Done.
[root@ivy ~]# /var/tmp/iperf3-static-sunos -c 10.200.200.202 -p 11845 -t 120 -P 4
Connecting to host 10.200.200.202, port 11845
[  5] local 10.200.200.201 port 57228 connected to 10.200.200.202 port 11845
[  7] local 10.200.200.201 port 43997 connected to 10.200.200.202 port 11845
[  9] local 10.200.200.201 port 48400 connected to 10.200.200.202 port 11845
[ 11] local 10.200.200.201 port 57597 connected to 10.200.200.202 port 11845
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec   621 MBytes  5.21 Gbits/sec
[  7]   0.00-1.00   sec   557 MBytes  4.67 Gbits/sec
[  9]   0.00-1.00   sec   610 MBytes  5.11 Gbits/sec
[ 11]   0.00-1.00   sec   636 MBytes  5.34 Gbits/sec
[SUM]   0.00-1.00   sec  2.37 GBytes  20.3 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   1.00-2.00   sec   688 MBytes  5.77 Gbits/sec
[  7]   1.00-2.00   sec   612 MBytes  5.13 Gbits/sec
[  9]   1.00-2.00   sec   669 MBytes  5.61 Gbits/sec
[ 11]   1.00-2.00   sec   699 MBytes  5.86 Gbits/sec
[SUM]   1.00-2.00   sec  2.61 GBytes  22.4 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   2.00-3.00   sec   686 MBytes  5.76 Gbits/sec
[  7]   2.00-3.00   sec   614 MBytes  5.15 Gbits/sec
[  9]   2.00-3.00   sec   672 MBytes  5.63 Gbits/sec
[ 11]   2.00-3.00   sec   697 MBytes  5.85 Gbits/sec
[SUM]   2.00-3.00   sec  2.61 GBytes  22.4 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
<snip>
[  5] 118.00-119.00 sec   687 MBytes  5.76 Gbits/sec
[  7] 118.00-119.00 sec   618 MBytes  5.19 Gbits/sec
[  9] 118.00-119.00 sec   668 MBytes  5.61 Gbits/sec
[ 11] 118.00-119.00 sec   718 MBytes  6.02 Gbits/sec
[SUM] 118.00-119.00 sec  2.63 GBytes  22.6 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5] 119.00-120.00 sec   681 MBytes  5.71 Gbits/sec
[  7] 119.00-120.00 sec   613 MBytes  5.14 Gbits/sec
[  9] 119.00-120.00 sec   663 MBytes  5.56 Gbits/sec
[ 11] 119.00-120.00 sec   713 MBytes  5.98 Gbits/sec
[SUM] 119.00-120.00 sec  2.61 GBytes  22.4 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-120.00 sec  80.3 GBytes  5.75 Gbits/sec                  sender
[  5]   0.00-120.00 sec  80.3 GBytes  5.75 Gbits/sec                  receiver
[  7]   0.00-120.00 sec  71.8 GBytes  5.14 Gbits/sec                  sender
[  7]   0.00-120.00 sec  71.8 GBytes  5.14 Gbits/sec                  receiver
[  9]   0.00-120.00 sec  78.1 GBytes  5.59 Gbits/sec                  sender
[  9]   0.00-120.00 sec  78.1 GBytes  5.59 Gbits/sec                  receiver
[ 11]   0.00-120.00 sec  82.8 GBytes  5.92 Gbits/sec                  sender
[ 11]   0.00-120.00 sec  82.8 GBytes  5.92 Gbits/sec                  receiver
[SUM]   0.00-120.00 sec   313 GBytes  22.4 Gbits/sec                  sender
[SUM]   0.00-120.00 sec   313 GBytes  22.4 Gbits/sec                  receiver

iperf Done.
[root@ivy ~]# /var/tmp/iperf3-static-sunos -c 10.200.200.202 -p 11845 -t 120 -P 4
Connecting to host 10.200.200.202, port 11845
[  5] local 10.200.200.201 port 59691 connected to 10.200.200.202 port 11845
[  7] local 10.200.200.201 port 60754 connected to 10.200.200.202 port 11845
[  9] local 10.200.200.201 port 39711 connected to 10.200.200.202 port 11845
[ 11] local 10.200.200.201 port 57824 connected to 10.200.200.202 port 11845
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec   413 MBytes  3.47 Gbits/sec
[  7]   0.00-1.00   sec   734 MBytes  6.16 Gbits/sec
[  9]   0.00-1.00   sec   397 MBytes  3.33 Gbits/sec
[ 11]   0.00-1.00   sec   830 MBytes  6.97 Gbits/sec
[SUM]   0.00-1.00   sec  2.32 GBytes  19.9 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   1.00-2.00   sec   459 MBytes  3.85 Gbits/sec
[  7]   1.00-2.00   sec   841 MBytes  7.06 Gbits/sec
[  9]   1.00-2.00   sec   445 MBytes  3.73 Gbits/sec
[ 11]   1.00-2.00   sec   856 MBytes  7.18 Gbits/sec
[SUM]   1.00-2.00   sec  2.54 GBytes  21.8 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   2.00-3.00   sec   463 MBytes  3.88 Gbits/sec
[  7]   2.00-3.00   sec   845 MBytes  7.09 Gbits/sec
[  9]   2.00-3.00   sec   448 MBytes  3.76 Gbits/sec
[ 11]   2.00-3.00   sec   864 MBytes  7.24 Gbits/sec
[SUM]   2.00-3.00   sec  2.56 GBytes  22.0 Gbits/sec
<snip>
[  5] 118.00-119.00 sec   535 MBytes  4.49 Gbits/sec
[  7] 118.00-119.00 sec   794 MBytes  6.67 Gbits/sec
[  9] 118.00-119.00 sec   515 MBytes  4.32 Gbits/sec
[ 11] 118.00-119.00 sec   448 MBytes  3.76 Gbits/sec
[SUM] 118.00-119.00 sec  2.24 GBytes  19.2 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5] 119.00-120.00 sec   537 MBytes  4.51 Gbits/sec
[  7] 119.00-120.00 sec   803 MBytes  6.74 Gbits/sec
[  9] 119.00-120.00 sec   516 MBytes  4.33 Gbits/sec
[ 11] 119.00-120.00 sec   453 MBytes  3.80 Gbits/sec
[SUM] 119.00-120.00 sec  2.26 GBytes  19.4 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-120.00 sec  61.6 GBytes  4.41 Gbits/sec                  sender
[  5]   0.00-120.00 sec  61.6 GBytes  4.41 Gbits/sec                  receiver
[  7]   0.00-120.00 sec  93.6 GBytes  6.70 Gbits/sec                  sender
[  7]   0.00-120.00 sec  93.6 GBytes  6.70 Gbits/sec                  receiver
[  9]   0.00-120.00 sec  59.3 GBytes  4.24 Gbits/sec                  sender
[  9]   0.00-120.00 sec  59.3 GBytes  4.24 Gbits/sec                  receiver
[ 11]   0.00-120.00 sec  63.5 GBytes  4.54 Gbits/sec                  sender
[ 11]   0.00-120.00 sec  63.5 GBytes  4.54 Gbits/sec                  receiver
[SUM]   0.00-120.00 sec   278 GBytes  19.9 Gbits/sec                  sender
[SUM]   0.00-120.00 sec   278 GBytes  19.9 Gbits/sec                  receiver

iperf Done.
[root@ivy ~]# /var/tmp/iperf3-static-sunos -c 10.200.200.202 -p 11845 -t 120 -P 4
Connecting to host 10.200.200.202, port 11845
[  5] local 10.200.200.201 port 46703 connected to 10.200.200.202 port 11845
[  7] local 10.200.200.201 port 57035 connected to 10.200.200.202 port 11845
[  9] local 10.200.200.201 port 54622 connected to 10.200.200.202 port 11845
[ 11] local 10.200.200.201 port 51194 connected to 10.200.200.202 port 11845
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec   482 MBytes  4.04 Gbits/sec
[  7]   0.00-1.00   sec  1.08 GBytes  9.30 Gbits/sec
[  9]   0.00-1.00   sec   346 MBytes  2.90 Gbits/sec
[ 11]   0.00-1.00   sec   817 MBytes  6.85 Gbits/sec
[SUM]   0.00-1.00   sec  2.69 GBytes  23.1 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   1.00-2.00   sec   505 MBytes  4.23 Gbits/sec
[  7]   1.00-2.00   sec  1.15 GBytes  9.85 Gbits/sec
[  9]   1.00-2.00   sec   367 MBytes  3.08 Gbits/sec
[ 11]   1.00-2.00   sec   877 MBytes  7.36 Gbits/sec
[SUM]   1.00-2.00   sec  2.85 GBytes  24.5 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   2.00-3.00   sec   514 MBytes  4.31 Gbits/sec
[  7]   2.00-3.00   sec  1.15 GBytes  9.88 Gbits/sec
[  9]   2.00-3.00   sec   364 MBytes  3.05 Gbits/sec
[ 11]   2.00-3.00   sec   857 MBytes  7.19 Gbits/sec
[SUM]   2.00-3.00   sec  2.84 GBytes  24.4 Gbits/sec
<snip>
[  5] 118.00-119.00 sec   517 MBytes  4.34 Gbits/sec
[  7] 118.00-119.00 sec  1.10 GBytes  9.47 Gbits/sec
[  9] 118.00-119.00 sec   362 MBytes  3.03 Gbits/sec
[ 11] 118.00-119.00 sec   845 MBytes  7.09 Gbits/sec
[SUM] 118.00-119.00 sec  2.79 GBytes  23.9 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5] 119.00-120.00 sec   510 MBytes  4.28 Gbits/sec
[  7] 119.00-120.00 sec  1.09 GBytes  9.35 Gbits/sec
[  9] 119.00-120.00 sec   364 MBytes  3.05 Gbits/sec
[ 11] 119.00-120.00 sec   860 MBytes  7.21 Gbits/sec
[SUM] 119.00-120.00 sec  2.78 GBytes  23.9 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-120.00 sec  60.3 GBytes  4.31 Gbits/sec                  sender
[  5]   0.00-120.00 sec  60.3 GBytes  4.31 Gbits/sec                  receiver
[  7]   0.00-120.00 sec   133 GBytes  9.49 Gbits/sec                  sender
[  7]   0.00-120.00 sec   133 GBytes  9.49 Gbits/sec                  receiver
[  9]   0.00-120.00 sec  42.7 GBytes  3.06 Gbits/sec                  sender
[  9]   0.00-120.00 sec  42.7 GBytes  3.06 Gbits/sec                  receiver
[ 11]   0.00-120.00 sec   100 GBytes  7.19 Gbits/sec                  sender
[ 11]   0.00-120.00 sec   100 GBytes  7.19 Gbits/sec                  receiver
[SUM]   0.00-120.00 sec   336 GBytes  24.0 Gbits/sec                  sender
[SUM]   0.00-120.00 sec   336 GBytes  24.0 Gbits/sec                  receiver

iperf Done.

Both platforms are getting the same transfer rates.

The single thread runs take a while on both platforms to get up to ~16 Gbits/sec, which I at first thought was due to the congestion window growing slowly, but was revealed by connstat to actually be due to the RTT starting out at around 2000-3000 microseconds and then dropping after a while to 400 microseconds. I wrote some more about why this is happening in OS-7751.

#2

Updated by Electric Monk about 1 hour ago

  • Status changed from New to Closed
  • % Done changed from 0 to 100

git commit c12492cf73149aa0aa845af5d59966b0eb5aa910

commit  c12492cf73149aa0aa845af5d59966b0eb5aa910
Author: Sebastien Roy <seb@delphix.com>
Date:   2019-08-19T22:32:46.000Z

    11546 Track TCP round-trip time in nanoseconds
    Portions contributed by: Cody Peter Mello <cody.mello@joyent.com>
    Portions contributed by: Brandon Baker <bbaker@delphix.com>
    Reviewed by: Jason King <jason.king@joyent.com>
    Reviewed by: Robert Mustacchi <rm@joyent.com>
    Reviewed by: Dan McDonald <danmcd@joyent.com>
    Approved by: Richard Lowe <richlowe@richlowe.net>

Also available in: Atom PDF