Project

General

Profile

Bug #10409

ipf sometimes freezes RFC 1323 transfers

Added by Dan McDonald 8 months ago. Updated 8 months ago.

Status:
Closed
Priority:
Normal
Assignee:
Category:
networking
Start date:
2019-02-15
Due date:
% Done:

100%

Estimated time:
Difficulty:
Hard
Tags:
needs-triage

Description

Discovered in SmartOS https://smartos.org/bugview/OS-7586

Something is triggering packet drops, or at least one packet drop, and it's observable by seeing fr_tcpinwindow() (in ipf) returning 0 when processing a packet with data (i.e. not just an ACK). The attached DTrace script will print data on fr_tcpinwindow() returning 0. If you see a packet of non-zero size here, it indicates this problem and a connection is hung.

Examining more recent ipfilter updates (such as the one proposed by an illumos community member to be ported into illumos), it appears the TCP state-maintenance checks have some flaws. There's a commit in FreeBSD:

https://github.com/freebsd/freebsd/commit/c636cf4e7104540ba66c035229138c1f5f52c374

That addressed a sabotage of RFC 1323 window-scaling. PARTS of this fix appeared in illumos already thanks to:

https://github.com/illumos/illumos-gate/commit/f17d2b412cd592f6085d3a1002cac662768ec1ae

but not ALL of them. This bug will address the full integration of the FreeBSD fix above to match what was already in illumos back from the early days of OpenSolaris.


Files

ipf-tcpinwindow.d (904 Bytes) ipf-tcpinwindow.d Dan McDonald, 2019-02-15 09:03 PM

History

#1

Updated by Dan McDonald 8 months ago

With the proposed fix in place, packets for large transfers are much smoother (observed by snoop -o /dev/null and watching the counter not stall every n-1000 packets), and the DTrace script produces NO OUTPUT.

#2

Updated by Dan McDonald 8 months ago

Additional internal testing from Joyent's Todd Whiteman:

"Testing in nightly-1 using a platform with Dan's fixes solved the docker pull issues I was seeing - 25 successful docker pulls of the holzi/quine-relay (1.5GB) image."

#3

Updated by Electric Monk 8 months ago

  • Status changed from New to Closed
  • % Done changed from 0 to 100

git commit 6a6ded8a9716c81d429d2d916745909354d93fee

commit  6a6ded8a9716c81d429d2d916745909354d93fee
Author: Dan McDonald <danmcd@joyent.com>
Date:   2019-02-20T21:34:13.000Z

    10409 ipf sometimes freezes RFC 1323 transfers
    Reviewed by: Jason King <jbk@joyent.com>
    Reviewed by: Andy Fiddaman <andy@omniosce.org>
    Reviewed by: Cody Peter Mello <melloc@writev.io>
    Reviewed by: Gergő Doma <domag02@gmail.com>
    Approved by: Robert Mustacchi <rm@joyent.com>

Also available in: Atom PDF