Project

General

Profile

Actions

Feature #5011

closed

fast localhost connections choke in TCP TIME_WAIT

Added by Robert Mustacchi about 7 years ago. Updated about 7 years ago.

Status:
Closed
Priority:
Normal
Assignee:
Category:
networking
Start date:
2014-07-16
Due date:
% Done:

100%

Estimated time:
Difficulty:
Medium
Tags:
Gerrit CR:

Description

Brendan's initial investigation:

When performing frequent localhost connections, the available range (by default, 32k, from tcp_smallest_anon_port to tcp_largest_anon_port) becomes stuck in TCP TIME_WAIT (tcp_time_wait_interval) for 60 seconds. Even though this is localhost (TIME_WAIT, WHY?). Linux does not suffer this issue.

Linux does have tunables to recycle TCP connections quickly (net.ipv4.tcp_tw_recycle, net.ipv4.tcp_tw_reuse), which sound related, but are set to zero. (Perhaps the localhost path is just special-cased.)

This is typically hit in two scenarios: localhost benchmarking, and when different application level components are run from the same SmartMachine.

Connection rate testing SmartOS:

[root@headnode (bh1-kvm4:0) /zones/brendan]# node load-server.js 5
Listening at http://127.0.0.1:8080/
Summary every 5 secs
Connections/sec: 2301
Connections/sec: 2981.6
Connections/sec: 1235
Connections/sec: 36
Connections/sec: 0
Connections/sec: 0
Connections/sec: 0
Connections/sec: 0
Connections/sec: 0
Connections/sec: 0
Connections/sec: 0
Connections/sec: 0
Connections/sec: 563.2
Connections/sec: 2471.8
Connections/sec: 2639.8
Connections/sec: 817.4
Connections/sec: 61.4
Connections/sec: 0
Connections/sec: 0
Connections/sec: 0
Connections/sec: 0
Connections/sec: 0
Connections/sec: 0
Connections/sec: 0
Connections/sec: 7
Connections/sec: 1081.2
Connections/sec: 2339.6
Connections/sec: 2399.6
Connections/sec: 665
Connections/sec: 61.2
Connections/sec: 0
Connections/sec: 0
Connections/sec: 0
Connections/sec: 0
Connections/sec: 0
Connections/sec: 0
Connections/sec: 3.6
Connections/sec: 454
Connections/sec: 1272.8
Connections/sec: 2260.2
Connections/sec: 2083.6
Connections/sec: 431.8
Connections/sec: 47.6
Connections/sec: 0
Connections/sec: 0
Connections/sec: 0
[...time passes...]
Connections/sec: 2.8
Connections/sec: 39.8
Connections/sec: 254.6
Connections/sec: 651
Connections/sec: 1111.6
Connections/sec: 1598.4
Connections/sec: 1569.8
Connections/sec: 904.6
Connections/sec: 291.2
Connections/sec: 104.6
Connections/sec: 23.8
Connections/sec: 1.2
Connections/sec: 2
Connections/sec: 19.8
Connections/sec: 161.8
Connections/sec: 489.2
Connections/sec: 897.4
Connections/sec: 1393.8
Connections/sec: 1594.2
Connections/sec: 1210.6
Connections/sec: 549.2
Connections/sec: 174

yuk.

[root@headnode (bh1-kvm4:0) /zones/brendan]# netstat -an | wc -l
   32854
[root@headnode (bh1-kvm4:0) /zones/brendan]# netstat -an
[...]
127.0.0.1.48347      127.0.0.1.8080       130880      0 139264      0 TIME_WAIT
127.0.0.1.54234      127.0.0.1.8080       130880      0 139264      0 TIME_WAIT
127.0.0.1.39607      127.0.0.1.8080       130880      0 139264      0 TIME_WAIT
127.0.0.1.59519      127.0.0.1.8080       130880      0 139264      0 TIME_WAIT
127.0.0.1.43772      127.0.0.1.8080       130880      0 139264      0 TIME_WAIT
127.0.0.1.45439      127.0.0.1.8080       130880      0 139264      0 TIME_WAIT
127.0.0.1.53384      127.0.0.1.8080       130880      0 139264      0 TIME_WAIT
127.0.0.1.39155      127.0.0.1.8080       130880      0 139264      0 TIME_WAIT
127.0.0.1.43643      127.0.0.1.8080       130880      0 139264      0 TIME_WAIT
127.0.0.1.55752      127.0.0.1.8080       130880      0 139264      0 TIME_WAIT
127.0.0.1.62010      127.0.0.1.8080       130880      0 139264      0 TIME_WAIT
[...]

Linux:

admin@bh1-kvm3:~$ node load-server.js 5
Listening at http://127.0.0.1:8080/
Summary every 5 secs
Connections/sec: 1541.8
Connections/sec: 2348.4
Connections/sec: 2258.4
Connections/sec: 2363
Connections/sec: 2335.8
Connections/sec: 2305.2
Connections/sec: 2294.8
Connections/sec: 2296.8
Connections/sec: 2433.2
Connections/sec: 2234.2
Connections/sec: 2231.2
Connections/sec: 2285.6
Connections/sec: 2347.8
Connections/sec: 2952
Connections/sec: 2989.2
Connections/sec: 3005
Connections/sec: 3013.4
Connections/sec: 3001
Connections/sec: 2979
Connections/sec: 3023.4
Connections/sec: 3033.6
Connections/sec: 3019.4
Connections/sec: 2458
Connections/sec: 2505.4
Connections/sec: 2346.6
Connections/sec: 2321.4
Connections/sec: 2295.2
Connections/sec: 2309.6
Connections/sec: 2367.8
Connections/sec: 2323.6
Connections/sec: 2224.6
Connections/sec: 2329
Connections/sec: 2338.4
Connections/sec: 2313
Connections/sec: 2270.6
Connections/sec: 2288.8
Connections/sec: 2276.2
Connections/sec: 2296.4
Connections/sec: 2253.4
Connections/sec: 2302.2
Connections/sec: 2318.8
Connections/sec: 2261.6

--

We opted here to special case localhost for time wait considerations. This is a frequent issue from customer's trying to benchmark one thing and in reality benchmarking another.

--

The following is the server Brendan used:

var http = require('http');
var conns = 0;

http.createServer(function (req, res) {
    res.writeHead(200, {'Content-Type': 'text/plain'});
    res.end('G\\'Day\\n');
    conns++;
}).listen(8199, "0.0.0.0");

function status(interval) {
    setTimeout(function() {
        console.log('Connections/sec: ' + conns / interval);
        conns = 0;
        status(interval);
    }, interval * 1000);
}

console.log('Listening at http://127.0.0.1:8199/');

if (process.argv.length > 2) {
    var interval = process.argv[2];
    console.log('Summary every ' + interval + ' secs');
    status(interval);
}

The following is the client:

var mod_http = require('http');
var uri = '/';
var concurrency = 10;

function get(hostname, port, uri)
{
    mod_http.get({ host: hostname, port: port, path: uri },
        function (response) {
        return (get(hostname, port, uri));
        }
    ).on('error', function() { get(hostname, port, uri) });
}

function main()
{
    var server, parts, hostname, port;

    if (process.argv.length <= 2) {
        console.error('USAGE: node load-client.js <server>[:<port>]');
        process.exit(1);
    }

    server = process.argv[2];
    parts = server.split(':');
    hostname = parts[0];
    port = parts[1] || '80';

    console.log('Lanching load at http://%s:%s%s', hostname, port, uri);

    get(hostname, port, uri);
}

main();
Actions #1

Updated by Electric Monk about 7 years ago

  • Status changed from New to Closed

git commit 81b60dac72578dc6fa8b5be0b69548c8ebba1da8

commit  81b60dac72578dc6fa8b5be0b69548c8ebba1da8
Author: Jerry Jelinek <jerry.jelinek@joyent.com>
Date:   2014-07-23T15:28:03.000Z

    5011 fast localhost connections choke in TCP TIME_WAIT
    Reviewed by: Robert Mustacchi <rm@joyent.com>
    Reviewed by: Dan McDonald <danmcd@omniti.com>
    Approved by: Gordon Ross <gwr@nexenta.com>

Actions

Also available in: Atom PDF