Actions
Bug #11546
closed
Track TCP round-trip time in nanoseconds
Status:
Closed
Priority:
Normal
Assignee:
-
Category:
networking
Start date:
Due date:
% Done:
100%
Estimated time:
Difficulty:
Medium
Tags:
Gerrit CR:
External Bug:
Description
As part of upstreaming Delphix's connstat(1M) changes, we'll want to pull in their updates to TCP to use nanosecond timestamps for RTT instead of microseconds. This includes changes made in DLPX-42721 and DLPX-43064. This change will also make TCP stop using echoed timestamps for calculating RTT; when committed, we can also close out #8725.
This work was initially done by Sebastien Roy and Brandon Baker at Delphix, with some further changes made by Cody Mello at Joyent in OS-7334.
Updated by Cody Mello about 4 years ago
These changes allow for measuring a more precise RTT, but that shouldn't have any impact on the performance. To verify this, I ran some iperf tests between two systems with 40GiB Chelsio NICs. On a recent SmartOS PI that doesn't include these changes (joyent_20190328T010321Z):
[root@ivy ~]# /var/tmp/iperf3-static-sunos -c 10.200.200.202 -p 11845 -t 120 Connecting to host 10.200.200.202, port 11845 [ 5] local 10.200.200.201 port 63240 connected to 10.200.200.202 port 11845 [ ID] Interval Transfer Bitrate [ 5] 0.00-1.00 sec 425 MBytes 3.56 Gbits/sec [ 5] 1.00-2.00 sec 693 MBytes 5.81 Gbits/sec [ 5] 2.00-3.00 sec 450 MBytes 3.78 Gbits/sec [ 5] 3.00-4.00 sec 658 MBytes 5.52 Gbits/sec [ 5] 4.00-5.00 sec 455 MBytes 3.82 Gbits/sec [ 5] 5.00-6.00 sec 450 MBytes 3.78 Gbits/sec [ 5] 6.00-7.00 sec 1.91 GBytes 16.4 Gbits/sec [ 5] 7.00-8.00 sec 452 MBytes 3.79 Gbits/sec [ 5] 8.00-9.00 sec 478 MBytes 4.01 Gbits/sec [ 5] 9.00-10.00 sec 450 MBytes 3.77 Gbits/sec [ 5] 10.00-11.00 sec 1.17 GBytes 10.0 Gbits/sec [ 5] 11.00-12.00 sec 635 MBytes 5.33 Gbits/sec [ 5] 12.00-13.00 sec 572 MBytes 4.79 Gbits/sec <snip> [ 5] 100.00-101.00 sec 1.94 GBytes 16.6 Gbits/sec [ 5] 101.00-102.00 sec 1.93 GBytes 16.5 Gbits/sec [ 5] 102.00-103.00 sec 1.94 GBytes 16.6 Gbits/sec [ 5] 103.00-104.00 sec 1.94 GBytes 16.6 Gbits/sec [ 5] 104.00-105.00 sec 1.92 GBytes 16.5 Gbits/sec [ 5] 105.00-106.00 sec 1.93 GBytes 16.6 Gbits/sec [ 5] 106.00-107.00 sec 1.93 GBytes 16.6 Gbits/sec [ 5] 107.00-108.00 sec 1.92 GBytes 16.5 Gbits/sec [ 5] 108.00-109.00 sec 1.93 GBytes 16.6 Gbits/sec [ 5] 109.00-110.00 sec 1.92 GBytes 16.5 Gbits/sec [ 5] 110.00-111.00 sec 1.92 GBytes 16.5 Gbits/sec [ 5] 111.00-112.00 sec 1.92 GBytes 16.5 Gbits/sec [ 5] 112.00-113.00 sec 1.92 GBytes 16.5 Gbits/sec [ 5] 113.00-114.00 sec 1.93 GBytes 16.6 Gbits/sec [ 5] 114.00-115.00 sec 1.92 GBytes 16.5 Gbits/sec [ 5] 115.00-116.00 sec 1.92 GBytes 16.5 Gbits/sec [ 5] 116.00-117.00 sec 1.91 GBytes 16.4 Gbits/sec [ 5] 117.00-118.00 sec 1.91 GBytes 16.4 Gbits/sec [ 5] 118.00-119.00 sec 1.92 GBytes 16.5 Gbits/sec [ 5] 119.00-120.00 sec 1.93 GBytes 16.6 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate [ 5] 0.00-120.00 sec 157 GBytes 11.2 Gbits/sec sender [ 5] 0.00-120.00 sec 157 GBytes 11.2 Gbits/sec receiver iperf Done. [root@ivy ~]# /var/tmp/iperf3-static-sunos -c 10.200.200.202 -p 11845 -t 120 Connecting to host 10.200.200.202, port 11845 [ 5] local 10.200.200.201 port 53375 connected to 10.200.200.202 port 11845 [ ID] Interval Transfer Bitrate [ 5] 0.00-1.00 sec 421 MBytes 3.53 Gbits/sec [ 5] 1.00-2.00 sec 1.17 GBytes 10.1 Gbits/sec [ 5] 2.00-3.00 sec 844 MBytes 7.08 Gbits/sec [ 5] 3.00-4.00 sec 516 MBytes 4.33 Gbits/sec [ 5] 4.00-5.00 sec 653 MBytes 5.47 Gbits/sec [ 5] 5.00-6.00 sec 712 MBytes 5.97 Gbits/sec [ 5] 6.00-7.00 sec 1.92 GBytes 16.5 Gbits/sec [ 5] 7.00-8.00 sec 1.96 GBytes 16.8 Gbits/sec [ 5] 8.00-9.00 sec 1.96 GBytes 16.8 Gbits/sec [ 5] 9.00-10.00 sec 1.95 GBytes 16.8 Gbits/sec [ 5] 10.00-11.00 sec 1.97 GBytes 16.9 Gbits/sec [ 5] 11.00-12.00 sec 1.97 GBytes 17.0 Gbits/sec [ 5] 12.00-13.00 sec 1.98 GBytes 17.0 Gbits/sec [ 5] 13.00-14.00 sec 1.98 GBytes 17.0 Gbits/sec <snip> [ 5] 100.00-101.00 sec 1.97 GBytes 16.9 Gbits/sec [ 5] 101.00-102.00 sec 1.97 GBytes 16.9 Gbits/sec [ 5] 102.00-103.00 sec 1.97 GBytes 16.9 Gbits/sec [ 5] 103.00-104.00 sec 1.97 GBytes 16.9 Gbits/sec [ 5] 104.00-105.00 sec 1.97 GBytes 16.9 Gbits/sec [ 5] 105.00-106.00 sec 1.97 GBytes 17.0 Gbits/sec [ 5] 106.00-107.00 sec 1.97 GBytes 16.9 Gbits/sec [ 5] 107.00-108.00 sec 1.95 GBytes 16.8 Gbits/sec [ 5] 108.00-109.00 sec 1.96 GBytes 16.9 Gbits/sec [ 5] 109.00-110.00 sec 1.96 GBytes 16.9 Gbits/sec [ 5] 110.00-111.00 sec 1.97 GBytes 17.0 Gbits/sec [ 5] 111.00-112.00 sec 1.98 GBytes 17.0 Gbits/sec [ 5] 112.00-113.00 sec 1.96 GBytes 16.9 Gbits/sec [ 5] 113.00-114.00 sec 1.95 GBytes 16.8 Gbits/sec [ 5] 114.00-115.00 sec 1.95 GBytes 16.8 Gbits/sec [ 5] 115.00-116.00 sec 1.95 GBytes 16.8 Gbits/sec [ 5] 116.00-117.00 sec 1.95 GBytes 16.8 Gbits/sec [ 5] 117.00-118.00 sec 1.95 GBytes 16.8 Gbits/sec [ 5] 118.00-119.00 sec 1.95 GBytes 16.8 Gbits/sec [ 5] 119.00-120.00 sec 1.96 GBytes 16.8 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate [ 5] 0.00-120.00 sec 228 GBytes 16.3 Gbits/sec sender [ 5] 0.00-120.00 sec 228 GBytes 16.3 Gbits/sec receiver iperf Done. [root@ivy ~]# /var/tmp/iperf3-static-sunos -c 10.200.200.202 -p 11845 -t 120 -P 4 Connecting to host 10.200.200.202, port 11845 [ 5] local 10.200.200.201 port 64322 connected to 10.200.200.202 port 11845 [ 7] local 10.200.200.201 port 35299 connected to 10.200.200.202 port 11845 [ 9] local 10.200.200.201 port 51000 connected to 10.200.200.202 port 11845 [ 11] local 10.200.200.201 port 36272 connected to 10.200.200.202 port 11845 [ ID] Interval Transfer Bitrate [ 5] 0.00-1.00 sec 637 MBytes 5.34 Gbits/sec [ 7] 0.00-1.00 sec 835 MBytes 7.01 Gbits/sec [ 9] 0.00-1.00 sec 417 MBytes 3.50 Gbits/sec [ 11] 0.00-1.00 sec 630 MBytes 5.28 Gbits/sec [SUM] 0.00-1.00 sec 2.46 GBytes 21.1 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ 5] 1.00-2.00 sec 659 MBytes 5.53 Gbits/sec [ 7] 1.00-2.00 sec 734 MBytes 6.15 Gbits/sec [ 9] 1.00-2.00 sec 655 MBytes 5.50 Gbits/sec [ 11] 1.00-2.00 sec 612 MBytes 5.13 Gbits/sec [SUM] 1.00-2.00 sec 2.60 GBytes 22.3 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ 5] 2.00-3.00 sec 626 MBytes 5.25 Gbits/sec [ 7] 2.00-3.00 sec 693 MBytes 5.81 Gbits/sec [ 9] 2.00-3.00 sec 620 MBytes 5.20 Gbits/sec [ 11] 2.00-3.00 sec 864 MBytes 7.25 Gbits/sec [SUM] 2.00-3.00 sec 2.74 GBytes 23.5 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - <snip> [ 5] 118.00-119.00 sec 616 MBytes 5.17 Gbits/sec [ 7] 118.00-119.00 sec 655 MBytes 5.49 Gbits/sec [ 9] 118.00-119.00 sec 602 MBytes 5.05 Gbits/sec [ 11] 118.00-119.00 sec 736 MBytes 6.17 Gbits/sec [SUM] 118.00-119.00 sec 2.55 GBytes 21.9 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ 5] 119.00-120.00 sec 614 MBytes 5.16 Gbits/sec [ 7] 119.00-120.00 sec 688 MBytes 5.78 Gbits/sec [ 9] 119.00-120.00 sec 599 MBytes 5.03 Gbits/sec [ 11] 119.00-120.00 sec 764 MBytes 6.42 Gbits/sec [SUM] 119.00-120.00 sec 2.60 GBytes 22.4 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate [ 5] 0.00-120.00 sec 71.0 GBytes 5.08 Gbits/sec sender [ 5] 0.00-120.00 sec 71.0 GBytes 5.08 Gbits/sec receiver [ 7] 0.00-120.00 sec 80.6 GBytes 5.77 Gbits/sec sender [ 7] 0.00-120.00 sec 80.6 GBytes 5.77 Gbits/sec receiver [ 9] 0.00-120.00 sec 70.4 GBytes 5.04 Gbits/sec sender [ 9] 0.00-120.00 sec 70.4 GBytes 5.04 Gbits/sec receiver [ 11] 0.00-120.00 sec 89.2 GBytes 6.38 Gbits/sec sender [ 11] 0.00-120.00 sec 89.2 GBytes 6.38 Gbits/sec receiver [SUM] 0.00-120.00 sec 311 GBytes 22.3 Gbits/sec sender [SUM] 0.00-120.00 sec 311 GBytes 22.3 Gbits/sec receiver iperf Done. [root@ivy ~]# /var/tmp/iperf3-static-sunos -c 10.200.200.202 -p 11845 -t 120 -P 4 Connecting to host 10.200.200.202, port 11845 [ 5] local 10.200.200.201 port 41045 connected to 10.200.200.202 port 11845 [ 7] local 10.200.200.201 port 51532 connected to 10.200.200.202 port 11845 [ 9] local 10.200.200.201 port 55819 connected to 10.200.200.202 port 11845 [ 11] local 10.200.200.201 port 49547 connected to 10.200.200.202 port 11845 [ ID] Interval Transfer Bitrate [ 5] 0.00-1.00 sec 740 MBytes 6.21 Gbits/sec [ 7] 0.00-1.00 sec 582 MBytes 4.88 Gbits/sec [ 9] 0.00-1.00 sec 554 MBytes 4.65 Gbits/sec [ 11] 0.00-1.00 sec 408 MBytes 3.42 Gbits/sec [SUM] 0.00-1.00 sec 2.23 GBytes 19.2 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ 5] 1.00-2.00 sec 721 MBytes 6.05 Gbits/sec [ 7] 1.00-2.00 sec 584 MBytes 4.90 Gbits/sec [ 9] 1.00-2.00 sec 724 MBytes 6.07 Gbits/sec [ 11] 1.00-2.00 sec 490 MBytes 4.11 Gbits/sec [SUM] 1.00-2.00 sec 2.46 GBytes 21.1 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - <snip> [ 5] 118.00-119.00 sec 525 MBytes 4.41 Gbits/sec [ 7] 118.00-119.00 sec 900 MBytes 7.55 Gbits/sec [ 9] 118.00-119.00 sec 459 MBytes 3.85 Gbits/sec [ 11] 118.00-119.00 sec 818 MBytes 6.86 Gbits/sec [SUM] 118.00-119.00 sec 2.64 GBytes 22.7 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ 5] 119.00-120.00 sec 502 MBytes 4.21 Gbits/sec [ 7] 119.00-120.00 sec 915 MBytes 7.68 Gbits/sec [ 9] 119.00-120.00 sec 430 MBytes 3.61 Gbits/sec [ 11] 119.00-120.00 sec 863 MBytes 7.24 Gbits/sec [SUM] 119.00-120.00 sec 2.65 GBytes 22.7 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate [ 5] 0.00-120.00 sec 63.6 GBytes 4.56 Gbits/sec sender [ 5] 0.00-120.00 sec 63.6 GBytes 4.56 Gbits/sec receiver [ 7] 0.00-120.00 sec 98.2 GBytes 7.03 Gbits/sec sender [ 7] 0.00-120.00 sec 98.2 GBytes 7.03 Gbits/sec receiver [ 9] 0.00-120.00 sec 54.8 GBytes 3.92 Gbits/sec sender [ 9] 0.00-120.00 sec 54.8 GBytes 3.92 Gbits/sec receiver [ 11] 0.00-120.00 sec 93.8 GBytes 6.71 Gbits/sec sender [ 11] 0.00-120.00 sec 93.8 GBytes 6.71 Gbits/sec receiver [SUM] 0.00-120.00 sec 310 GBytes 22.2 Gbits/sec sender [SUM] 0.00-120.00 sec 310 GBytes 22.2 Gbits/sec receiver iperf Done.
On a SmartOS platform built with these changes (and #11547):
[root@ivy ~]# /var/tmp/iperf3-static-sunos -c 10.200.200.202 -p 11845 -t 120 Connecting to host 10.200.200.202, port 11845 [ 5] local 10.200.200.201 port 45276 connected to 10.200.200.202 port 11845 [ ID] Interval Transfer Bitrate [ 5] 0.00-1.00 sec 755 MBytes 6.34 Gbits/sec [ 5] 1.00-2.00 sec 1.09 GBytes 9.37 Gbits/sec [ 5] 2.00-3.00 sec 451 MBytes 3.78 Gbits/sec [ 5] 3.00-4.00 sec 451 MBytes 3.79 Gbits/sec [ 5] 4.00-5.00 sec 450 MBytes 3.77 Gbits/sec [ 5] 5.00-6.00 sec 1.16 GBytes 10.0 Gbits/sec [ 5] 6.00-7.00 sec 859 MBytes 7.20 Gbits/sec [ 5] 7.00-8.00 sec 619 MBytes 5.19 Gbits/sec [ 5] 8.00-9.00 sec 984 MBytes 8.26 Gbits/sec [ 5] 9.00-10.00 sec 735 MBytes 6.17 Gbits/sec [ 5] 10.00-11.00 sec 916 MBytes 7.68 Gbits/sec [ 5] 11.00-12.00 sec 1.59 GBytes 13.6 Gbits/sec [ 5] 12.00-13.00 sec 974 MBytes 8.17 Gbits/sec [ 5] 13.00-14.00 sec 759 MBytes 6.37 Gbits/sec [ 5] 14.00-15.00 sec 1.74 GBytes 14.9 Gbits/sec [ 5] 15.00-16.00 sec 1.16 GBytes 9.93 Gbits/sec [ 5] 16.00-17.00 sec 453 MBytes 3.80 Gbits/sec [ 5] 17.00-18.00 sec 952 MBytes 7.99 Gbits/sec [ 5] 18.00-19.00 sec 546 MBytes 4.58 Gbits/sec [ 5] 19.00-20.00 sec 1.18 GBytes 10.1 Gbits/sec [ 5] 20.00-21.00 sec 1.41 GBytes 12.1 Gbits/sec [ 5] 21.00-22.00 sec 1.98 GBytes 17.0 Gbits/sec [ 5] 22.00-23.00 sec 1.99 GBytes 17.1 Gbits/sec [ 5] 23.00-24.00 sec 1.98 GBytes 17.0 Gbits/sec [ 5] 24.00-25.00 sec 1.98 GBytes 17.0 Gbits/sec [ 5] 25.00-26.00 sec 1.99 GBytes 17.1 Gbits/sec <snip> [ 5] 100.00-101.00 sec 1.92 GBytes 16.5 Gbits/sec [ 5] 101.00-102.00 sec 1.93 GBytes 16.6 Gbits/sec [ 5] 102.00-103.00 sec 1.92 GBytes 16.5 Gbits/sec [ 5] 103.00-104.00 sec 1.92 GBytes 16.5 Gbits/sec [ 5] 104.00-105.00 sec 1.93 GBytes 16.5 Gbits/sec [ 5] 105.00-106.00 sec 1.92 GBytes 16.5 Gbits/sec [ 5] 106.00-107.00 sec 1.92 GBytes 16.5 Gbits/sec [ 5] 107.00-108.00 sec 1.92 GBytes 16.5 Gbits/sec [ 5] 108.00-109.00 sec 1.92 GBytes 16.5 Gbits/sec [ 5] 109.00-110.00 sec 1.92 GBytes 16.5 Gbits/sec [ 5] 110.00-111.00 sec 1.94 GBytes 16.7 Gbits/sec [ 5] 111.00-112.00 sec 1.94 GBytes 16.7 Gbits/sec [ 5] 112.00-113.00 sec 1.94 GBytes 16.7 Gbits/sec [ 5] 113.00-114.00 sec 1.94 GBytes 16.7 Gbits/sec [ 5] 114.00-115.00 sec 1.94 GBytes 16.7 Gbits/sec [ 5] 115.00-116.00 sec 1.92 GBytes 16.5 Gbits/sec [ 5] 116.00-117.00 sec 1.92 GBytes 16.5 Gbits/sec [ 5] 117.00-118.00 sec 1.92 GBytes 16.5 Gbits/sec [ 5] 118.00-119.00 sec 1.92 GBytes 16.5 Gbits/sec [ 5] 119.00-120.00 sec 1.92 GBytes 16.5 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate [ 5] 0.00-120.00 sec 202 GBytes 14.5 Gbits/sec sender [ 5] 0.00-120.00 sec 202 GBytes 14.5 Gbits/sec receiver iperf Done. [root@ivy ~]# /var/tmp/iperf3-static-sunos -c 10.200.200.202 -p 11845 -t 120 Connecting to host 10.200.200.202, port 11845 [ 5] local 10.200.200.201 port 33259 connected to 10.200.200.202 port 11845 [ ID] Interval Transfer Bitrate [ 5] 0.00-1.00 sec 506 MBytes 4.25 Gbits/sec [ 5] 1.00-2.00 sec 1.14 GBytes 9.79 Gbits/sec [ 5] 2.00-3.00 sec 640 MBytes 5.37 Gbits/sec [ 5] 3.00-4.00 sec 450 MBytes 3.78 Gbits/sec [ 5] 4.00-5.00 sec 515 MBytes 4.32 Gbits/sec [ 5] 5.00-6.00 sec 689 MBytes 5.78 Gbits/sec [ 5] 6.00-7.00 sec 486 MBytes 4.07 Gbits/sec [ 5] 7.00-8.00 sec 659 MBytes 5.53 Gbits/sec [ 5] 8.00-9.00 sec 573 MBytes 4.81 Gbits/sec [ 5] 9.00-10.00 sec 1017 MBytes 8.53 Gbits/sec [ 5] 10.00-11.00 sec 453 MBytes 3.80 Gbits/sec [ 5] 11.00-12.00 sec 460 MBytes 3.86 Gbits/sec [ 5] 12.00-13.00 sec 1.58 GBytes 13.6 Gbits/sec [ 5] 13.00-14.00 sec 1.10 GBytes 9.46 Gbits/sec [ 5] 14.00-15.00 sec 1.97 GBytes 16.9 Gbits/sec [ 5] 15.00-16.00 sec 1.97 GBytes 16.9 Gbits/sec [ 5] 16.00-17.00 sec 1.96 GBytes 16.9 Gbits/sec [ 5] 17.00-18.00 sec 1.96 GBytes 16.9 Gbits/sec [ 5] 18.00-19.00 sec 1.97 GBytes 16.9 Gbits/sec [ 5] 19.00-20.00 sec 1.96 GBytes 16.9 Gbits/sec <snip> [ 5] 100.00-101.00 sec 1.97 GBytes 16.9 Gbits/sec [ 5] 101.00-102.00 sec 1.98 GBytes 17.0 Gbits/sec [ 5] 102.00-103.00 sec 1.98 GBytes 17.0 Gbits/sec [ 5] 103.00-104.00 sec 1.98 GBytes 17.1 Gbits/sec [ 5] 104.00-105.00 sec 1.98 GBytes 17.0 Gbits/sec [ 5] 105.00-106.00 sec 1.99 GBytes 17.1 Gbits/sec [ 5] 106.00-107.00 sec 1.99 GBytes 17.1 Gbits/sec [ 5] 107.00-108.00 sec 1.96 GBytes 16.9 Gbits/sec [ 5] 108.00-109.00 sec 1.95 GBytes 16.8 Gbits/sec [ 5] 109.00-110.00 sec 1.96 GBytes 16.9 Gbits/sec [ 5] 110.00-111.00 sec 1.96 GBytes 16.8 Gbits/sec [ 5] 111.00-112.00 sec 1.95 GBytes 16.8 Gbits/sec [ 5] 112.00-113.00 sec 1.95 GBytes 16.7 Gbits/sec [ 5] 113.00-114.00 sec 1.96 GBytes 16.8 Gbits/sec [ 5] 114.00-115.00 sec 1.97 GBytes 16.9 Gbits/sec [ 5] 115.00-116.00 sec 1.97 GBytes 16.9 Gbits/sec [ 5] 116.00-117.00 sec 1.96 GBytes 16.8 Gbits/sec [ 5] 117.00-118.00 sec 1.96 GBytes 16.8 Gbits/sec [ 5] 118.00-119.00 sec 1.96 GBytes 16.8 Gbits/sec [ 5] 119.00-120.00 sec 1.96 GBytes 16.8 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate [ 5] 0.00-120.00 sec 218 GBytes 15.6 Gbits/sec sender [ 5] 0.00-120.00 sec 218 GBytes 15.6 Gbits/sec receiver iperf Done. [root@ivy ~]# /var/tmp/iperf3-static-sunos -c 10.200.200.202 -p 11845 -t 120 -P 4 Connecting to host 10.200.200.202, port 11845 [ 5] local 10.200.200.201 port 57228 connected to 10.200.200.202 port 11845 [ 7] local 10.200.200.201 port 43997 connected to 10.200.200.202 port 11845 [ 9] local 10.200.200.201 port 48400 connected to 10.200.200.202 port 11845 [ 11] local 10.200.200.201 port 57597 connected to 10.200.200.202 port 11845 [ ID] Interval Transfer Bitrate [ 5] 0.00-1.00 sec 621 MBytes 5.21 Gbits/sec [ 7] 0.00-1.00 sec 557 MBytes 4.67 Gbits/sec [ 9] 0.00-1.00 sec 610 MBytes 5.11 Gbits/sec [ 11] 0.00-1.00 sec 636 MBytes 5.34 Gbits/sec [SUM] 0.00-1.00 sec 2.37 GBytes 20.3 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ 5] 1.00-2.00 sec 688 MBytes 5.77 Gbits/sec [ 7] 1.00-2.00 sec 612 MBytes 5.13 Gbits/sec [ 9] 1.00-2.00 sec 669 MBytes 5.61 Gbits/sec [ 11] 1.00-2.00 sec 699 MBytes 5.86 Gbits/sec [SUM] 1.00-2.00 sec 2.61 GBytes 22.4 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ 5] 2.00-3.00 sec 686 MBytes 5.76 Gbits/sec [ 7] 2.00-3.00 sec 614 MBytes 5.15 Gbits/sec [ 9] 2.00-3.00 sec 672 MBytes 5.63 Gbits/sec [ 11] 2.00-3.00 sec 697 MBytes 5.85 Gbits/sec [SUM] 2.00-3.00 sec 2.61 GBytes 22.4 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - <snip> [ 5] 118.00-119.00 sec 687 MBytes 5.76 Gbits/sec [ 7] 118.00-119.00 sec 618 MBytes 5.19 Gbits/sec [ 9] 118.00-119.00 sec 668 MBytes 5.61 Gbits/sec [ 11] 118.00-119.00 sec 718 MBytes 6.02 Gbits/sec [SUM] 118.00-119.00 sec 2.63 GBytes 22.6 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ 5] 119.00-120.00 sec 681 MBytes 5.71 Gbits/sec [ 7] 119.00-120.00 sec 613 MBytes 5.14 Gbits/sec [ 9] 119.00-120.00 sec 663 MBytes 5.56 Gbits/sec [ 11] 119.00-120.00 sec 713 MBytes 5.98 Gbits/sec [SUM] 119.00-120.00 sec 2.61 GBytes 22.4 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate [ 5] 0.00-120.00 sec 80.3 GBytes 5.75 Gbits/sec sender [ 5] 0.00-120.00 sec 80.3 GBytes 5.75 Gbits/sec receiver [ 7] 0.00-120.00 sec 71.8 GBytes 5.14 Gbits/sec sender [ 7] 0.00-120.00 sec 71.8 GBytes 5.14 Gbits/sec receiver [ 9] 0.00-120.00 sec 78.1 GBytes 5.59 Gbits/sec sender [ 9] 0.00-120.00 sec 78.1 GBytes 5.59 Gbits/sec receiver [ 11] 0.00-120.00 sec 82.8 GBytes 5.92 Gbits/sec sender [ 11] 0.00-120.00 sec 82.8 GBytes 5.92 Gbits/sec receiver [SUM] 0.00-120.00 sec 313 GBytes 22.4 Gbits/sec sender [SUM] 0.00-120.00 sec 313 GBytes 22.4 Gbits/sec receiver iperf Done. [root@ivy ~]# /var/tmp/iperf3-static-sunos -c 10.200.200.202 -p 11845 -t 120 -P 4 Connecting to host 10.200.200.202, port 11845 [ 5] local 10.200.200.201 port 59691 connected to 10.200.200.202 port 11845 [ 7] local 10.200.200.201 port 60754 connected to 10.200.200.202 port 11845 [ 9] local 10.200.200.201 port 39711 connected to 10.200.200.202 port 11845 [ 11] local 10.200.200.201 port 57824 connected to 10.200.200.202 port 11845 [ ID] Interval Transfer Bitrate [ 5] 0.00-1.00 sec 413 MBytes 3.47 Gbits/sec [ 7] 0.00-1.00 sec 734 MBytes 6.16 Gbits/sec [ 9] 0.00-1.00 sec 397 MBytes 3.33 Gbits/sec [ 11] 0.00-1.00 sec 830 MBytes 6.97 Gbits/sec [SUM] 0.00-1.00 sec 2.32 GBytes 19.9 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ 5] 1.00-2.00 sec 459 MBytes 3.85 Gbits/sec [ 7] 1.00-2.00 sec 841 MBytes 7.06 Gbits/sec [ 9] 1.00-2.00 sec 445 MBytes 3.73 Gbits/sec [ 11] 1.00-2.00 sec 856 MBytes 7.18 Gbits/sec [SUM] 1.00-2.00 sec 2.54 GBytes 21.8 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ 5] 2.00-3.00 sec 463 MBytes 3.88 Gbits/sec [ 7] 2.00-3.00 sec 845 MBytes 7.09 Gbits/sec [ 9] 2.00-3.00 sec 448 MBytes 3.76 Gbits/sec [ 11] 2.00-3.00 sec 864 MBytes 7.24 Gbits/sec [SUM] 2.00-3.00 sec 2.56 GBytes 22.0 Gbits/sec <snip> [ 5] 118.00-119.00 sec 535 MBytes 4.49 Gbits/sec [ 7] 118.00-119.00 sec 794 MBytes 6.67 Gbits/sec [ 9] 118.00-119.00 sec 515 MBytes 4.32 Gbits/sec [ 11] 118.00-119.00 sec 448 MBytes 3.76 Gbits/sec [SUM] 118.00-119.00 sec 2.24 GBytes 19.2 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ 5] 119.00-120.00 sec 537 MBytes 4.51 Gbits/sec [ 7] 119.00-120.00 sec 803 MBytes 6.74 Gbits/sec [ 9] 119.00-120.00 sec 516 MBytes 4.33 Gbits/sec [ 11] 119.00-120.00 sec 453 MBytes 3.80 Gbits/sec [SUM] 119.00-120.00 sec 2.26 GBytes 19.4 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate [ 5] 0.00-120.00 sec 61.6 GBytes 4.41 Gbits/sec sender [ 5] 0.00-120.00 sec 61.6 GBytes 4.41 Gbits/sec receiver [ 7] 0.00-120.00 sec 93.6 GBytes 6.70 Gbits/sec sender [ 7] 0.00-120.00 sec 93.6 GBytes 6.70 Gbits/sec receiver [ 9] 0.00-120.00 sec 59.3 GBytes 4.24 Gbits/sec sender [ 9] 0.00-120.00 sec 59.3 GBytes 4.24 Gbits/sec receiver [ 11] 0.00-120.00 sec 63.5 GBytes 4.54 Gbits/sec sender [ 11] 0.00-120.00 sec 63.5 GBytes 4.54 Gbits/sec receiver [SUM] 0.00-120.00 sec 278 GBytes 19.9 Gbits/sec sender [SUM] 0.00-120.00 sec 278 GBytes 19.9 Gbits/sec receiver iperf Done. [root@ivy ~]# /var/tmp/iperf3-static-sunos -c 10.200.200.202 -p 11845 -t 120 -P 4 Connecting to host 10.200.200.202, port 11845 [ 5] local 10.200.200.201 port 46703 connected to 10.200.200.202 port 11845 [ 7] local 10.200.200.201 port 57035 connected to 10.200.200.202 port 11845 [ 9] local 10.200.200.201 port 54622 connected to 10.200.200.202 port 11845 [ 11] local 10.200.200.201 port 51194 connected to 10.200.200.202 port 11845 [ ID] Interval Transfer Bitrate [ 5] 0.00-1.00 sec 482 MBytes 4.04 Gbits/sec [ 7] 0.00-1.00 sec 1.08 GBytes 9.30 Gbits/sec [ 9] 0.00-1.00 sec 346 MBytes 2.90 Gbits/sec [ 11] 0.00-1.00 sec 817 MBytes 6.85 Gbits/sec [SUM] 0.00-1.00 sec 2.69 GBytes 23.1 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ 5] 1.00-2.00 sec 505 MBytes 4.23 Gbits/sec [ 7] 1.00-2.00 sec 1.15 GBytes 9.85 Gbits/sec [ 9] 1.00-2.00 sec 367 MBytes 3.08 Gbits/sec [ 11] 1.00-2.00 sec 877 MBytes 7.36 Gbits/sec [SUM] 1.00-2.00 sec 2.85 GBytes 24.5 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ 5] 2.00-3.00 sec 514 MBytes 4.31 Gbits/sec [ 7] 2.00-3.00 sec 1.15 GBytes 9.88 Gbits/sec [ 9] 2.00-3.00 sec 364 MBytes 3.05 Gbits/sec [ 11] 2.00-3.00 sec 857 MBytes 7.19 Gbits/sec [SUM] 2.00-3.00 sec 2.84 GBytes 24.4 Gbits/sec <snip> [ 5] 118.00-119.00 sec 517 MBytes 4.34 Gbits/sec [ 7] 118.00-119.00 sec 1.10 GBytes 9.47 Gbits/sec [ 9] 118.00-119.00 sec 362 MBytes 3.03 Gbits/sec [ 11] 118.00-119.00 sec 845 MBytes 7.09 Gbits/sec [SUM] 118.00-119.00 sec 2.79 GBytes 23.9 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ 5] 119.00-120.00 sec 510 MBytes 4.28 Gbits/sec [ 7] 119.00-120.00 sec 1.09 GBytes 9.35 Gbits/sec [ 9] 119.00-120.00 sec 364 MBytes 3.05 Gbits/sec [ 11] 119.00-120.00 sec 860 MBytes 7.21 Gbits/sec [SUM] 119.00-120.00 sec 2.78 GBytes 23.9 Gbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate [ 5] 0.00-120.00 sec 60.3 GBytes 4.31 Gbits/sec sender [ 5] 0.00-120.00 sec 60.3 GBytes 4.31 Gbits/sec receiver [ 7] 0.00-120.00 sec 133 GBytes 9.49 Gbits/sec sender [ 7] 0.00-120.00 sec 133 GBytes 9.49 Gbits/sec receiver [ 9] 0.00-120.00 sec 42.7 GBytes 3.06 Gbits/sec sender [ 9] 0.00-120.00 sec 42.7 GBytes 3.06 Gbits/sec receiver [ 11] 0.00-120.00 sec 100 GBytes 7.19 Gbits/sec sender [ 11] 0.00-120.00 sec 100 GBytes 7.19 Gbits/sec receiver [SUM] 0.00-120.00 sec 336 GBytes 24.0 Gbits/sec sender [SUM] 0.00-120.00 sec 336 GBytes 24.0 Gbits/sec receiver iperf Done.
Both platforms are getting the same transfer rates.
The single thread runs take a while on both platforms to get up to ~16 Gbits/sec, which I at first thought was due to the congestion window growing slowly, but was revealed by connstat to actually be due to the RTT starting out at around 2000-3000 microseconds and then dropping after a while to 400 microseconds. I wrote some more about why this is happening in OS-7751.
Updated by Electric Monk about 4 years ago
- Status changed from New to Closed
- % Done changed from 0 to 100
git commit c12492cf73149aa0aa845af5d59966b0eb5aa910
commit c12492cf73149aa0aa845af5d59966b0eb5aa910
Author: Sebastien Roy <seb@delphix.com>
Date: 2019-08-19T22:32:46.000Z
11546 Track TCP round-trip time in nanoseconds
Portions contributed by: Cody Peter Mello <cody.mello@joyent.com>
Portions contributed by: Brandon Baker <bbaker@delphix.com>
Reviewed by: Jason King <jason.king@joyent.com>
Reviewed by: Robert Mustacchi <rm@joyent.com>
Reviewed by: Dan McDonald <danmcd@joyent.com>
Approved by: Richard Lowe <richlowe@richlowe.net>
Actions