illumos gate

The current implementation of the IPPROTO_TCP/TCP_INIT_CWND only affects a socket before accept() and connect(). This is quite limiting and unnecessary. There is a Google patch for Linux that implements this stuff correctly there and we should adopt their approach. They allow (via ioctl, I recommend keeping this a setsockopt) a modification of the initial window until data is first sent on the socket.

Specifically, this allows for following:

p = socket
bind(p) // http web server
listen(p)
setsockopt(p, IPPROTO_TCP, TCP_INIT_CWND, &10...)
c = accept(p)
// up to here currently works, the rest is limited by the current implementation
request = read(c)
if(someanalysis(request))
setsockopt(c, IPPROTO_TCP, TCP_INIT_CWND, &2...)
write(c, response)

This use-case demonstrates adjusting the initial congestion window before data send on the socket to enable different treatment of clients based on network information and request information (like user-agent indicating mobile browser).