Project

General

Profile

Bug #10969

SMB server listener stops after a SYN-ACK flood

Added by Gordon Ross 8 days ago. Updated 4 days ago.

Status:
Closed
Priority:
Normal
Assignee:
Category:
-
Start date:
2019-05-14
Due date:
% Done:

100%

Estimated time:
Difficulty:
Medium
Tags:

Description

Data from the customer indicates the server was exiting after the TCP 3-way handshake. As such this is an ECONNABORTED (error # 130 - see customers notes below.) Rather than try to restart for any ksocket_accept() error returned, it's probably safer to only try to restart if the server exits due to this one error.

Good news, I spoke to the tester who was running the nessus test and he said that he has seen the ksocket_accept fail before. (we have added some logging to the smb_server_listener so it will print something in the log when the ksocket_socket accept fails).

He was able to cause the smb_server_listener thread to exit when running a performance test. The return code from ksocket_accept is 130.

I checked mdb and the confirmed that the following thread is not running anymore:

ffffff0030d9fc40 fffffffffbc2e3c0                0   0  60 ffffff07273785f0
  PC: _resume_from_idle+0xf4    THREAD: smb_thread_entry_point()

  stack pointer for thread ffffff0030d9fc40: ffffff0030d9f900
  [ ffffff0030d9f900 _resume_from_idle+0xf4() ]
    swtch+0x145()
    cv_wait+0x61()
    cv_wait_sig_swap_core+0x2aa()
    cv_wait_sig_swap+0x18()
    so_acceptq_dequeue_locked+0x63()
    so_acceptq_dequeue+0x40()
    so_accept+0x9a()
    socket_accept+0x22()
    ksocket_accept+0x116()
    smb_server_listener+0x12d()
    smb_thread_entry_point+0x51()
    thread_start+8()

The test involves simulating very high SMB2 tree connections ( 20,000 per second) using SwiftTest. It doesn’t happen every time, but if we run it 2-3 times we will see the issue.

History

#1

Updated by Gordon Ross 5 days ago

  • Description updated (diff)
  • Status changed from New to In Progress
#2

Updated by Gordon Ross 5 days ago

  • Description updated (diff)
#3

Updated by Gordon Ross 5 days ago

  • Description updated (diff)
#4

Updated by Gordon Ross 5 days ago

The fix restarts the listener in the case of an ECONNABORTED which results from TCP SYN-ACK flood. Previously, if the listener exited due to this it would not restart.

Testing: Simulate SYN/ACK flood (and SwiftTest at the customer)
Fix in production since mid 2016

#5

Updated by Joshua Clulow 4 days ago

  • Description updated (diff)
  • Tags deleted (needs-triage)
#6

Updated by Electric Monk 4 days ago

  • Status changed from In Progress to Closed
  • % Done changed from 0 to 100

git commit 41bd85101cd90603ab3220d835461fe851a81fed

commit  41bd85101cd90603ab3220d835461fe851a81fed
Author: kcrowenex <kevin.crowe@nexenta.com>
Date:   2019-05-19T23:21:13.000Z

    10969 SMB server listener stops after a SYN-ACK flood
    Reviewed by: Gordon Ross <gwr@nexenta.com>
    Reviewed by: Matt Barden <matt.barden@nexenta.com>
    Approved by: Joshua M. Clulow <josh@sysmgr.org>

Also available in: Atom PDF