Project

General

Profile

Actions

Bug #7814

closed

Accept TCP keepalives with missing timestamps

Added by Prakash Surya over 4 years ago. Updated over 4 years ago.

Status:
Closed
Priority:
Normal
Assignee:
Category:
networking
Start date:
2017-01-27
Due date:
% Done:

100%

Estimated time:
Difficulty:
Medium
Tags:
needs-triage
Gerrit CR:

Description

Running DelphixOS with the following illumos commit:

commit 1f183ba0b0be3e10202501aa3740753df6512804
Author: Lauri Tirkkonen <>
Date: Wed Apr 15 16:30:46 2015 +0300
5850 tcp timestamping behavior changed mid-connection

We've seen Microsoft SQL Server needlessly reset its JBDC connection.

The problem stems from the fact that some TCP implementations (e.g.
Microsoft's) do not put timestamps on keepalive segments. The above
referenced illumos commit implements the following "SHOULD" behavior
from RFC-7323:

Once TSopt has been successfully negotiated, that is both <SYN> and
<SYN,ACK> contain TSopt, the TSopt MUST be sent in every non-<RST>
segment for the duration of the connection, and SHOULD be sent in an
<RST> segment (see Section 5.2 for details). The TCP SHOULD remember
this state by setting a flag, referred to as Snd.TS.OK, to one. If a
non-<RST> segment is received without a TSopt, a TCP SHOULD silently
drop the segment. A TCP MUST NOT abort a TCP connection because any
segment lacks an expected TSopt.

For the full RFC, see here: https://tools.ietf.org/html/rfc7323

As a result of implementing that part of the RFC specification, if the
use of timestamps was negotiated, all packets that do not have
timestamps are dropped; including keepalives.

A further complication is that legacy TCP stacks may send a garbage byte
in a keepalive packet, so the segment length in this case may be 0 or 1.

The following DTrace one-liner can be used to check for dropped packets
due to missing timestamps:

dtrace -n 'tcp:::droppedtimestamp { stack(); }'

While some may argue the "bug" is actually in these other TCP stacks
that send keepalives without timestamps, and rather than us cooperating
with these broken TCP implementations those stacks should
be fixed, that approach isn't practical for a couple of reasons:

1. We cannot force the maintainers of those other TCP implementations
to make the changes to their code, such that they'd properly adhere
to the RFC.
2. Even if we could force them to fix their code, the broken
implementations may still be found "in the wild", as there's no
gaurantee that the updated/fixed code would percolate to all of the
already affected systems.

Thus, the only practical approach for us to solve the problem (and the
approach taken by this patch), is to accept keepalives with missing
timestamps.


Related issues

Related to illumos gate - Bug #5850: tcp timestamping behavior changed mid-connectionClosedLauri Tirkkonen2015-04-17

Actions
Actions #1

Updated by Electric Monk over 4 years ago

  • Status changed from New to Closed
  • % Done changed from 0 to 100

git commit 143b26e0f0139cacf020594bec08f738845897d3

commit  143b26e0f0139cacf020594bec08f738845897d3
Author: Steve Gonczi <steve.gonczi@delphix.com>
Date:   2017-02-01T05:08:09.000Z

    7814 Accept TCP keepalives with missing timestamps
    Reviewed by: Sebastien Roy <sebastien.roy@delphix.com>
    Reviewed by: Dan Kimmel <dan.kimmel@delphix.com>
    Reviewed by: Pavel Zakharov <pavel.zakharov@delphix.com>
    Reviewed by: Robert Mustacchi <rm@joyent.com>
    Reviewed by: Dan McDonald <danmcd@omniti.com>
    Approved by: Gordon Ross <gordon.w.ross@gmail.com>

Actions #2

Updated by Marcel Telka over 3 years ago

  • Related to Bug #5850: tcp timestamping behavior changed mid-connection added
Actions

Also available in: Atom PDF