Project

General

Profile

Bug #11554

Want TCP_CONGESTION socket option

Added by Cody Mello about 1 year ago. Updated about 1 year ago.

Status:
Closed
Priority:
Normal
Assignee:
Category:
networking
Start date:
Due date:
% Done:

100%

Estimated time:
Difficulty:
Medium
Tags:
Gerrit CR:

Description

As part of testing #11553, I wanted to use the -C flag with iperf, which selects a congestion control algorithm to use while it runs. On both FreeBSD and Linux, this works by calling setsockopt(fd, IPPROTO_TCP, TCP_CONGESTION, "algo", sizeof ("algo")). It would be nice if illumos supported this socket option.

Note that on Linux, it seems to be okay to pass in a length for the string that doesn't include the NUL terminating character. For example iperf gets away with passing in strlen().

#1

Updated by Cody Mello about 1 year ago

To test this change, I built iperf, which makes use of the TCP_CONGESTION socket option when it's present on the system:

cpm@utu ~/src % git clone https://github.com/esnet/iperf.git
Cloning into 'iperf'...
remote: Enumerating objects: 12, done.
remote: Counting objects: 100% (12/12), done.
remote: Compressing objects: 100% (10/10), done.
remote: Total 7619 (delta 3), reused 7 (delta 2), pack-reused 7607
Receiving objects: 100% (7619/7619), 11.01 MiB | 4.60 MiB/s, done.
Resolving deltas: 100% (5371/5371), done.
cpm@utu ~/src % cd iperf
cpm@utu ~/src/iperf : master % ./configure
checking for a BSD-compatible install... /usr/bin/install -c
checking whether build environment is sane... yes
checking for a thread-safe mkdir -p... /usr/gnu/bin/mkdir -p
checking for gawk... gawk
checking whether make sets $(MAKE)... yes
checking whether make supports nested variables... yes
<snip ...>
checking whether compiling and linking against OpenSSL works... yes
checking TCP_CONGESTION socket option... yes
checking IPv6 flowlabel support... no
checking for cpuset_setaffinity... no
checking for sched_setaffinity... no
checking for SetProcessAffinityMask... no
checking for daemon... yes
checking for sendfile... no
checking for getline... yes
checking SO_MAX_PACING_RATE socket option... no
checking for library containing clock_gettime... none required
checking for clock_gettime... yes
checking that generated files are newer than configure... done
configure: creating ./config.status
config.status: creating Makefile
config.status: creating src/Makefile
config.status: creating src/version.h
config.status: creating examples/Makefile
config.status: creating iperf3.spec
config.status: creating src/iperf_config.h
config.status: executing depfiles commands
config.status: executing libtool commands
cpm@utu ~/src/iperf : master % make
<snip ...>
cpm@utu ~/src/iperf : master % ./src/iperf3 -C cubic -c 37.153.108.173 -p 11845 -t 10 
Connecting to host 37.153.108.173, port 11845
[  5] local 10.88.88.201 port 51705 connected to 37.153.108.173 port 11845
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec   170 KBytes  1.39 Mbits/sec                  
[  5]   1.00-2.00   sec  3.06 MBytes  25.6 Mbits/sec                  
[  5]   2.00-3.00   sec  6.85 MBytes  57.5 Mbits/sec                  
[  5]   3.00-4.00   sec  6.62 MBytes  55.5 Mbits/sec                  
[  5]   4.00-5.00   sec  6.55 MBytes  54.9 Mbits/sec                  
[  5]   5.00-6.00   sec  6.70 MBytes  56.2 Mbits/sec                  
[  5]   6.00-7.00   sec  6.45 MBytes  54.1 Mbits/sec                  
[  5]   7.00-8.00   sec  6.46 MBytes  54.2 Mbits/sec                  
[  5]   8.00-9.00   sec  6.67 MBytes  56.0 Mbits/sec                  
[  5]   9.00-10.00  sec  6.74 MBytes  56.5 Mbits/sec                  
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.00  sec  56.3 MBytes  47.2 Mbits/sec                  sender
[  5]   0.00-10.00  sec  56.2 MBytes  47.2 Mbits/sec                  receiver

iperf Done.

Using truss, we can see that it's setting the socket option:

cpm@utu ~/src/iperf : master % truss -t setsockopt ./src/iperf3 -C cubic -c 37.153.108.173 -p 11845 -t 1
    Received signal #18, SIGCLD, in waitid() [caught]
      siginfo: SIGCLD CLD_EXITED pid=103916 status=0x0000
    Received signal #18, SIGCLD, in waitid() [caught]
      siginfo: SIGCLD CLD_EXITED pid=103919 status=0x0000
    Received signal #18, SIGCLD, in waitid() [caught]
      siginfo: SIGCLD CLD_EXITED pid=103922 status=0x0000
Connecting to host 37.153.108.173, port 11845
setsockopt(5, tcp, TCP_CONGESTION, 0x085392C0, 5, SOV_DEFAULT) = 0
[  5] local 10.88.88.201 port 55921 connected to 37.153.108.173 port 11845
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec   113 KBytes   926 Kbits/sec                  
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec   113 KBytes   926 Kbits/sec                  sender
[  5]   0.00-1.00   sec  63.6 KBytes   521 Kbits/sec                  receiver

iperf Done.
#2

Updated by Cody Mello about 1 year ago

If you try to set a nonexistent algorithm, it returns ENOENT, like Linux does:

cpm@utu ~/src/iperf : master % truss -t setsockopt ./src/iperf3 -C cubic3 -c 37.153.108.173 -p 11845 -t 1
    Received signal #18, SIGCLD, in waitid() [caught]
      siginfo: SIGCLD CLD_EXITED pid=103931 status=0x0000
    Received signal #18, SIGCLD, in waitid() [caught]
      siginfo: SIGCLD CLD_EXITED pid=103934 status=0x0000
    Received signal #18, SIGCLD, in waitid() [caught]
      siginfo: SIGCLD CLD_EXITED pid=103937 status=0x0000
Connecting to host 37.153.108.173, port 11845
setsockopt(5, tcp, TCP_CONGESTION, 0x08D922C0, 6, SOV_DEFAULT) Err#2 ENOENT
iperf3: error - unable to set TCP_CONGESTION: Supplied congestion control algorithm not supported on this host
cpm@utu ~/src/iperf : master % truss -t setsockopt ./src/iperf3 -C sunreno1 -c 37.153.108.173 -p 11845 -t 1
    Received signal #18, SIGCLD, in waitid() [caught]
      siginfo: SIGCLD CLD_EXITED pid=103945 status=0x0000
    Received signal #18, SIGCLD, in waitid() [caught]
      siginfo: SIGCLD CLD_EXITED pid=103948 status=0x0000
    Received signal #18, SIGCLD, in waitid() [caught]
      siginfo: SIGCLD CLD_EXITED pid=103951 status=0x0000
Connecting to host 37.153.108.173, port 11845
setsockopt(5, tcp, TCP_CONGESTION, 0x084402C0, 8, SOV_DEFAULT) Err#2 ENOENT
iperf3: error - unable to set TCP_CONGESTION: Supplied congestion control algorithm not supported on this host

For comparison with Linux:

cpm@enlil ~ % strace -e trace=setsockopt iperf3 -C cubic3 -c 37.153.108.173 -p 11845 -t 1
Connecting to host 37.153.108.173, port 11845
setsockopt(5, SOL_TCP, TCP_CONGESTION, "cubic3", 6) = -1 ENOENT (No such file or directory)
iperf3: error - unable to set TCP_CONGESTION: Supplied congestion control algorithm not supported on this host
+++ exited with 1 +++
#3

Updated by Cody Mello about 1 year ago

To verify that using getsockopt() with TCP_CONGESTION works, I modified the example TCP client program in tcp(7P) to contain the following lines:

char name[40] = ""; 
int namelen = 40; 
if (getsockopt(fd, IPPROTO_TCP, TCP_CONGESTION, name, &namelen) == -1) {
    perror("getsockopt(TCP_CONGESTION) failed");
    return (1);
}   
printf("using the \"%s\" cc algorithm\n", name);

Running the program:

cpm@utu ~/src % pfexec ipadm set-prop tcp -p congestion_control=cubic
cpm@utu ~/src % ./client ::1 8080                                    
using the "cubic" cc algorithm
hello
cpm@utu ~/src % pfexec ipadm set-prop tcp -p congestion_control=sunreno
cpm@utu ~/src % ./client ::1 8080                                      
using the "sunreno" cc algorithm
hello
cpm@utu ~/src % pfexec ipadm set-prop tcp -p congestion_control=newreno
cpm@utu ~/src % ./client ::1 8080                                      
using the "newreno" cc algorithm
hello
#4

Updated by Electric Monk about 1 year ago

  • Status changed from New to Closed
  • % Done changed from 0 to 100

git commit d49945110829673d27d215f4db010ac1d22a68de

commit  d49945110829673d27d215f4db010ac1d22a68de
Author: Cody Peter Mello <cody.mello@joyent.com>
Date:   2019-08-23T18:44:26.000Z

    11554 Want TCP_CONGESTION socket option
    Reviewed by: Yuri Pankov <yuri.pankov@nexenta.com>
    Approved by: Richard Lowe <richlowe@richlowe.net>

Also available in: Atom PDF