illumos gate

Running DelphixOS with the following illumos commit:

commit 1f183ba0b0be3e10202501aa3740753df6512804
    Author: Lauri Tirkkonen <lotheac@iki.fi>
    Date:   Wed Apr 15 16:30:46 2015 +0300

5850 tcp timestamping behavior changed mid-connection

We've seen Microsoft SQL Server needlessly reset its JBDC connection.

The problem stems from the fact that some TCP implementations (e.g.
Microsoft's) do not put timestamps on keepalive segments. The above
referenced illumos commit implements the following "SHOULD" behavior
from RFC-7323:

Once TSopt has been successfully negotiated, that is both <SYN> and
    <SYN,ACK> contain TSopt, the TSopt MUST be sent in every non-<RST>
    segment for the duration of the connection, and SHOULD be sent in an
    <RST> segment (see Section 5.2 for details).  The TCP SHOULD remember
    this state by setting a flag, referred to as Snd.TS.OK, to one.  If a
    non-<RST> segment is received without a TSopt, a TCP SHOULD silently
    drop the segment.  A TCP MUST NOT abort a TCP connection because any
    segment lacks an expected TSopt.

For the full RFC, see here: https://tools.ietf.org/html/rfc7323

As a result of implementing that part of the RFC specification, if the
use of timestamps was negotiated, all packets that do not have
timestamps are dropped; including keepalives.

A further complication is that legacy TCP stacks may send a garbage byte
in a keepalive packet, so the segment length in this case may be 0 or 1.

The following DTrace one-liner can be used to check for dropped packets
due to missing timestamps:

dtrace -n 'tcp:::droppedtimestamp { stack(); }'

While some may argue the "bug" is actually in these other TCP stacks
that send keepalives without timestamps, and rather than us cooperating
with these broken TCP implementations those stacks should
be fixed, that approach isn't practical for a couple of reasons:

1. We cannot force the maintainers of those other TCP implementations
     to make the changes to their code, such that they'd properly adhere
     to the RFC.

2. Even if we could force them to fix their code, the broken
     implementations may still be found "in the wild", as there's no
     gaurantee that the updated/fixed code would percolate to all of the
     already affected systems.

Thus, the only practical approach for us to solve the problem (and the
approach taken by this patch), is to accept keepalives with missing
timestamps.