Project

General

Profile

Feature #9531

Want netstat -u to show PIDs associated with sockets

Added by Andy Fiddaman over 1 year ago. Updated 2 days ago.

Status:
In Progress
Priority:
Normal
Assignee:
Category:
kernel
Start date:
2018-05-10
Due date:
% Done:

90%

Estimated time:
20.00 h
Difficulty:
Hard
Tags:

Description

Mohamed A. Khalfella <> did a lot of work on this in 2015/16 - see https://github.com/khalfella/illumos-gate/tree/ns01
It had a few reviews but has stagnated since.


Files

netstat-new.svg (284 KB) netstat-new.svg Andy Fiddaman, 2019-09-18 07:11 PM

History

#1

Updated by Andy Fiddaman about 1 month ago

  • Subject changed from Want netstat -u to show PIDs associated with sockets. to Want netstat -u to show PIDs associated with sockets
  • Status changed from New to In Progress
  • % Done changed from 60 to 90
#2

Updated by Andy Fiddaman about 1 month ago

There is a review for a proposed patch at https://illumos.org/rb/r/2311/

As part of this work I evaluated the proposed change by Mohamed. That change worked functionally well but testing showed up that it was using a lot of CPU time just keeping track of socket ownership, regardless of whether anyone was actually running netstat. As an example, the attached flamegraph was taken on a test server running apache httpd and the apache benchmarking tool (ab) and 17% of time was spent doing this housekeeping.

The solution that is currently out for review uses a different approach. The kernel passes back information about the vnode that underlies each socket in a new table and netstat correlates that with information gleaned from /proc. This approach means that the overhead is only incurred when running netstat -u. The approach is not as robust as the original - one of the pieces of information that is used for the cross-reference is the inode number of the underlying file. For a POSIX socket, this is synthesised from the memory address of the sonode structure in the kernel and so is not absolutely guaranteed to be unique. However, in all of the testing I have done for this change I have not seen a collision.

Sample output:

bloody% pfexec netstat -anu

UDP: IPv4
   Local Address        Remote Address      User    Pid      Command       State
-------------------- -------------------- -------- ------ -------------- ----------
      *.*                                 root        215 rcm_daemon     Unbound
      *.*                                 root        215 rcm_daemon     Unbound
      *.111                               daemon      567 rpcbind        Idle
      *.*                                 daemon      567 rpcbind        Unbound
      *.55278                             daemon      567 rpcbind        Idle
      *.111                               daemon      567 rpcbind        Idle
      *.*                                 daemon      567 rpcbind        Unbound
      *.62085                             daemon      567 rpcbind        Idle
      *.*                                 root        582 in.ndpd        Unbound

UDP: IPv6
   Local Address                     Remote Address                   User    Pid      Command       State      If
--------------------------------- --------------------------------- -------- ------ -------------- ---------- -----
      *.*                                                           root        215 rcm_daemon     Unbound
      *.111                                                         daemon      567 rpcbind        Idle
      *.*                                                           daemon      567 rpcbind        Unbound
      *.55278                                                       daemon      567 rpcbind        Idle
      *.*                                                           root        582 in.ndpd        Unbound

TCP: IPv4
   Local Address        Remote Address      User    Pid      Command     Swind  Send-Q Rwind  Recv-Q    State
-------------------- -------------------- -------- ------ -------------- ------ ------ ------ ------ -----------
      *.22                 *.*            root        551 sshd                0      0 128000      0 LISTEN
172.27.10.9.22       172.27.10.79.48460   af          554 sshd           134664      0 128872      0 ESTABLISHED
172.27.10.9.22       172.27.10.79.48460   root        552 sshd           134664      0 128872      0 ESTABLISHED
      *.111                *.*            daemon      567 rpcbind             0      0 128000      0 LISTEN
      *.*                  *.*            daemon      567 rpcbind             0      0 128000      0 IDLE
      *.111                *.*            daemon      567 rpcbind             0      0 128000      0 LISTEN
      *.*                  *.*            daemon      567 rpcbind             0      0 128000      0 IDLE
      *.48184              *.*            root        584 inetd               0      0 128000      0 LISTEN
      *.43966              *.*            root        584 inetd               0      0 128000      0 LISTEN

TCP: IPv6
   Local Address                     Remote Address                   User    Pid      Command     Swind  Send-Q Rwind  Recv-Q    State      If
--------------------------------- --------------------------------- -------- ------ -------------- ------ ------ ------ ------ ----------- -----
      *.22                              *.*                         root        551 sshd                0      0 128000      0 LISTEN
      *.111                             *.*                         daemon      567 rpcbind             0      0 128000      0 LISTEN
      *.*                               *.*                         daemon      567 rpcbind             0      0 128000      0 IDLE
      *.43966                           *.*                         root        584 inetd               0      0 128000      0 LISTEN

Active UNIX domain sockets
Address          Type       User     Pid    Command        Local Address                           Remote Address
---------------- ---------- -------- ------ -------------- --------------------------------------- ---------------------------------------
ffffff06f63d5410 dgram      root        582 in.ndpd        /var/run/in.ndpd_mib
ffffff06f63d57d0 stream-ord root        582 in.ndpd        /var/run/in.ndpd_ipadm
ffffff06f63d5b90 stream-ord af          554 sshd
ffffff06f48b3048 stream-ord root        552 sshd
ffffff06f48b3408 stream-ord af          554 sshd
ffffff06f48b3408 stream-ord root        552 sshd
ffffff06f480ab78 stream-ord root        339 hald                                                   /var/run/dbus/system_bus_socket
ffffff06f47e7030 stream-ord root        339 hald                                                   /var/run/hald/dbus-krtTyfGU6F
ffffff06f47e7030 stream-ord root        353 hald-addon-sto                                         /var/run/hald/dbus-krtTyfGU6F
ffffff06f47e73f0 stream-ord root        339 hald           /var/run/hald/dbus-krtTyfGU6F
ffffff06f47e73f0 stream-ord root        353 hald-addon-sto /var/run/hald/dbus-krtTyfGU6F
ffffff06f47e77b0 stream-ord root        584 inetd          /var/run/.inetd.uds
ffffff06f4726028 stream-ord root        276 dbus-daemon    /var/run/dbus/system_bus_socket
ffffff06f47267a8 stream-ord root        339 hald           /var/run/hald/dbus-krtTyfGU6F
ffffff06f47363e0 stream-ord root        349 hald-addon-sto                                         /var/run/hald/dbus-krtTyfGU6F
ffffff06f47367a0 stream-ord root        339 hald           /var/run/hald/dbus-krtTyfGU6F
ffffff06f4736b60 stream-ord root        339 hald           /var/run/hald/dbus-krtTyfGU6F
ffffff06f467e018 stream-ord root        346 hald-addon-acp                                         /var/run/hald/dbus-krtTyfGU6F
ffffff06f467e018 stream-ord root        345 hald-addon-cpu                                         /var/run/hald/dbus-krtTyfGU6F
ffffff06f467e3d8 stream-ord root        346 hald-addon-acp                                         /var/run/hald/dbus-krtTyfGU6F
ffffff06f467e3d8 stream-ord root        345 hald-addon-cpu                                         /var/run/hald/dbus-krtTyfGU6F
ffffff06f467e798 stream-ord root        339 hald           /var/run/hald/dbus-krtTyfGU6F
ffffff06f467eb58 stream-ord root        343 hald-addon-net                                         /var/run/hald/dbus-krtTyfGU6F
ffffff06f3500010 stream-ord root        339 hald           /var/run/hald/dbus-VAF61JEarP
ffffff06f3500010 stream-ord root        342 hald-runner    /var/run/hald/dbus-VAF61JEarP
ffffff06f35003d0 stream-ord root        339 hald                                                   /var/run/hald/dbus-VAF61JEarP
ffffff06f35003d0 stream-ord root        342 hald-runner                                            /var/run/hald/dbus-VAF61JEarP
ffffff06f3500790 stream-ord root        339 hald           /var/run/hald/dbus-krtTyfGU6F
ffffff06f3500b50 stream-ord root        276 dbus-daemon
ffffff06f269f008 stream-ord root        339 hald
ffffff06f269f008 stream-ord root        276 dbus-daemon
ffffff06f269f3c8 stream-ord root        339 hald           /var/run/hald/dbus-VAF61JEarP
ffffff06f269f3c8 stream-ord root        276 dbus-daemon    /var/run/hald/dbus-VAF61JEarP
ffffff06f269f788 stream-ord root        276 dbus-daemon    /var/run/dbus/system_bus_socket
bloody% pfexec netstat -anuv

UDP: IPv4
   Local Address        Remote Address      User    Pid     State       Command
-------------------- -------------------- -------- ------ ---------- --------------
      *.*                                 root        215 Unbound    /usr/lib/rcm/rcm_daemon
      *.*                                 root        215 Unbound    /usr/lib/rcm/rcm_daemon
      *.111                               daemon      567 Idle       /usr/sbin/rpcbind
      *.*                                 daemon      567 Unbound    /usr/sbin/rpcbind
      *.55278                             daemon      567 Idle       /usr/sbin/rpcbind
      *.111                               daemon      567 Idle       /usr/sbin/rpcbind
      *.*                                 daemon      567 Unbound    /usr/sbin/rpcbind
      *.62085                             daemon      567 Idle       /usr/sbin/rpcbind
      *.*                                 root        582 Unbound    /usr/lib/inet/in.ndpd

UDP: IPv6
   Local Address                     Remote Address                   User    Pid     State      If     Command
--------------------------------- --------------------------------- -------- ------ ---------- ----- --------------
      *.*                                                           root        215 Unbound          /usr/lib/rcm/rcm_daemon
      *.111                                                         daemon      567 Idle             /usr/sbin/rpcbind
      *.*                                                           daemon      567 Unbound          /usr/sbin/rpcbind
      *.55278                                                       daemon      567 Idle             /usr/sbin/rpcbind
      *.*                                                           root        582 Unbound          /usr/lib/inet/in.ndpd

TCP: IPv4
Local/Remote Address Swind   Snext     Suna   Rwind   Rnext     Rack    Rto   Mss     State      User    Pid      Command
-------------------- ------ -------- -------- ------ -------- -------- ----- ----- ----------- -------- ------ --------------
      *.22
      *.*                 0 00000000 00000000 128000 00000000 00000000  1125   536 LISTEN      root        551 /usr/sbin/sshd
172.27.10.9.22
172.27.10.79.48460   134664 810d599c 810d599c 128872 3429c4a2 3429c4a2   450  1448 ESTABLISHED af          554 /usr/sbin/sshd -R
172.27.10.9.22
172.27.10.79.48460   134664 810d599c 810d599c 128872 3429c4a2 3429c4a2   450  1448 ESTABLISHED root        552 /usr/sbin/sshd -R
      *.111
      *.*                 0 00000000 00000000 128000 00000000 00000000  1125  1220 LISTEN      daemon      567 /usr/sbin/rpcbind
      *.*
      *.*                 0 00000000 00000000 128000 00000000 00000000  1125  1220 IDLE        daemon      567 /usr/sbin/rpcbind
      *.111
      *.*                 0 00000000 00000000 128000 00000000 00000000  1125   536 LISTEN      daemon      567 /usr/sbin/rpcbind
      *.*
      *.*                 0 00000000 00000000 128000 00000000 00000000  1125   536 IDLE        daemon      567 /usr/sbin/rpcbind
      *.48184
      *.*                 0 00000000 00000000 128000 00000000 00000000  1125   536 LISTEN      root        584 /usr/lib/inet/inetd start
      *.43966
      *.*                 0 00000000 00000000 128000 00000000 00000000  1125  1220 LISTEN      root        584 /usr/lib/inet/inetd start

TCP: IPv6
Local/Remote Address              Swind   Snext     Suna   Rwind   Rnext     Rack    Rto   Mss     State      If    User    Pid      Command
--------------------------------- ------ -------- -------- ------ -------- -------- ----- ----- ----------- ----- -------- ------ --------------
      *.22
      *.*                              0 00000000 00000000 128000 00000000 00000000  1125  1220 LISTEN            root        551 /usr/sbin/sshd
      *.111
      *.*                              0 00000000 00000000 128000 00000000 00000000  1125  1220 LISTEN            daemon      567 /usr/sbin/rpcbind
      *.*
      *.*                              0 00000000 00000000 128000 00000000 00000000  1125  1220 IDLE              daemon      567 /usr/sbin/rpcbind
      *.43966
      *.*                              0 00000000 00000000 128000 00000000 00000000  1125  1220 LISTEN            root        584 /usr/lib/inet/inetd start

Active UNIX domain sockets
Address          Type       User     Pid    Local Address                           Remote Address                          Command
---------------- ---------- -------- ------ --------------------------------------- --------------------------------------- --------------
ffffff06f63d5410 dgram      root        582 /var/run/in.ndpd_mib                                                            /usr/lib/inet/in.ndpd
ffffff06f63d57d0 stream-ord root        582 /var/run/in.ndpd_ipadm                                                          /usr/lib/inet/in.ndpd
ffffff06f63d5b90 stream-ord af          554                                                                                 /usr/sbin/sshd -R
ffffff06f48b3048 stream-ord root        552                                                                                 /usr/sbin/sshd -R
ffffff06f48b3408 stream-ord af          554                                                                                 /usr/sbin/sshd -R
ffffff06f48b3408 stream-ord root        552                                                                                 /usr/sbin/sshd -R
ffffff06f480ab78 stream-ord root        339                                         /var/run/dbus/system_bus_socket         /usr/lib/hal/hald --daemon=yes
ffffff06f47e7030 stream-ord root        339                                         /var/run/hald/dbus-krtTyfGU6F           /usr/lib/hal/hald --daemon=yes
ffffff06f47e7030 stream-ord root        353                                         /var/run/hald/dbus-krtTyfGU6F           /usr/lib/hal/hald-addon-storage
ffffff06f47e73f0 stream-ord root        339 /var/run/hald/dbus-krtTyfGU6F                                                   /usr/lib/hal/hald --daemon=yes
ffffff06f47e73f0 stream-ord root        353 /var/run/hald/dbus-krtTyfGU6F                                                   /usr/lib/hal/hald-addon-storage
ffffff06f47e77b0 stream-ord root        584 /var/run/.inetd.uds                                                             /usr/lib/inet/inetd start
ffffff06f4726028 stream-ord root        276 /var/run/dbus/system_bus_socket                                                 /usr/lib/dbus-daemon --system
ffffff06f47267a8 stream-ord root        339 /var/run/hald/dbus-krtTyfGU6F                                                   /usr/lib/hal/hald --daemon=yes
ffffff06f47363e0 stream-ord root        349                                         /var/run/hald/dbus-krtTyfGU6F           /usr/lib/hal/hald-addon-storage
ffffff06f47367a0 stream-ord root        339 /var/run/hald/dbus-krtTyfGU6F                                                   /usr/lib/hal/hald --daemon=yes
ffffff06f4736b60 stream-ord root        339 /var/run/hald/dbus-krtTyfGU6F                                                   /usr/lib/hal/hald --daemon=yes
ffffff06f467e018 stream-ord root        346                                         /var/run/hald/dbus-krtTyfGU6F           /usr/lib/hal/hald-addon-acpi
ffffff06f467e018 stream-ord root        345                                         /var/run/hald/dbus-krtTyfGU6F           /usr/lib/hal/hald-addon-cpufreq
ffffff06f467e3d8 stream-ord root        346                                         /var/run/hald/dbus-krtTyfGU6F           /usr/lib/hal/hald-addon-acpi
ffffff06f467e3d8 stream-ord root        345                                         /var/run/hald/dbus-krtTyfGU6F           /usr/lib/hal/hald-addon-cpufreq
ffffff06f467e798 stream-ord root        339 /var/run/hald/dbus-krtTyfGU6F                                                   /usr/lib/hal/hald --daemon=yes
ffffff06f467eb58 stream-ord root        343                                         /var/run/hald/dbus-krtTyfGU6F           /usr/lib/hal/hald-addon-network-discovery
ffffff06f3500010 stream-ord root        339 /var/run/hald/dbus-VAF61JEarP                                                   /usr/lib/hal/hald --daemon=yes
ffffff06f3500010 stream-ord root        342 /var/run/hald/dbus-VAF61JEarP                                                   hald-runner
ffffff06f35003d0 stream-ord root        339                                         /var/run/hald/dbus-VAF61JEarP           /usr/lib/hal/hald --daemon=yes
ffffff06f35003d0 stream-ord root        342                                         /var/run/hald/dbus-VAF61JEarP           hald-runner
ffffff06f3500790 stream-ord root        339 /var/run/hald/dbus-krtTyfGU6F                                                   /usr/lib/hal/hald --daemon=yes
ffffff06f3500b50 stream-ord root        276                                                                                 /usr/lib/dbus-daemon --system
ffffff06f269f008 stream-ord root        339                                                                                 /usr/lib/hal/hald --daemon=yes
ffffff06f269f008 stream-ord root        276                                                                                 /usr/lib/dbus-daemon --system
ffffff06f269f3c8 stream-ord root        339 /var/run/hald/dbus-VAF61JEarP                                                   /usr/lib/hal/hald --daemon=yes
ffffff06f269f3c8 stream-ord root        276 /var/run/hald/dbus-VAF61JEarP                                                   /usr/lib/dbus-daemon --system
ffffff06f269f788 stream-ord root        276 /var/run/dbus/system_bus_socket                                                 /usr/lib/dbus-daemon --system
#3

Updated by Andy Fiddaman 2 days ago

This has been running in OmniOS for a while and has been tested in a variety of ways.

For testing that the output of netstat has not materially changed, apart from that column headings are not consistent across outputs, variants of the following script were used:

mkdir out

t()
{
        op="out/`echo $* | tr ' ' '_' | sed 's/^-//'`" 
        netstat "$@" > $op.new
        netstat~ "$@" > $op.old
        diff -u $op.{old,new} > $op.diff && rm -f $op.diff
        diff -uw $op.{old,new} > $op.diffw && rm -f $op.diffw
}

for f in inet inet6; do
        for p in tcp udp; do
                t -n -f $f -P $p
                t -n -f $f -P $p -v
                t -an -f $f -P $p
                t -an -f $f -P $p -v
                t -s -f $f -P $p
        done
done

t -in
t -D
t -rn
t -m
t -M

Other tests have involved using netstat -u to check that information was present for various protocols and families and for both POSIX and TLI/XTI sockets. netstat -u has been run in a loop on a busy system and during system shutdown. Test programs have been written that create TLI sockets in various states (unbound, listening, etc.) and the output check against the expected results.

Also available in: Atom PDF