Actions
Feature #9531
closed
Want netstat -u to show PIDs associated with sockets
Start date:
2018-05-10
Due date:
% Done:
100%
Estimated time:
20.00 h
Difficulty:
Hard
Tags:
Gerrit CR:
External Bug:
Description
Mohamed A. Khalfella <khalfella@gmail.com> did a lot of work on this in 2015/16 - see https://github.com/khalfella/illumos-gate/tree/ns01
It had a few reviews but has stagnated since.
Files
Updated by Andy Fiddaman over 3 years ago
- Subject changed from Want netstat -u to show PIDs associated with sockets. to Want netstat -u to show PIDs associated with sockets
- Status changed from New to In Progress
- % Done changed from 60 to 90
Updated by Andy Fiddaman over 3 years ago
- File netstat-new.svg netstat-new.svg added
There is a review for a proposed patch at https://illumos.org/rb/r/2311/
As part of this work I evaluated the proposed change by Mohamed. That change worked functionally well but testing showed up that it was using a lot of CPU time just keeping track of socket ownership, regardless of whether anyone was actually running netstat. As an example, the attached flamegraph was taken on a test server running apache httpd and the apache benchmarking tool (ab) and 17% of time was spent doing this housekeeping.
The solution that is currently out for review uses a different approach. The kernel passes back information about the vnode that underlies each socket in a new table and netstat correlates that with information gleaned from /proc. This approach means that the overhead is only incurred when running netstat -u. The approach is not as robust as the original - one of the pieces of information that is used for the cross-reference is the inode number of the underlying file. For a POSIX socket, this is synthesised from the memory address of the sonode structure in the kernel and so is not absolutely guaranteed to be unique. However, in all of the testing I have done for this change I have not seen a collision.
Sample output:
bloody% pfexec netstat -anu
UDP: IPv4
Local Address Remote Address User Pid Command State
-------------------- -------------------- -------- ------ -------------- ----------
*.* root 215 rcm_daemon Unbound
*.* root 215 rcm_daemon Unbound
*.111 daemon 567 rpcbind Idle
*.* daemon 567 rpcbind Unbound
*.55278 daemon 567 rpcbind Idle
*.111 daemon 567 rpcbind Idle
*.* daemon 567 rpcbind Unbound
*.62085 daemon 567 rpcbind Idle
*.* root 582 in.ndpd Unbound
UDP: IPv6
Local Address Remote Address User Pid Command State If
--------------------------------- --------------------------------- -------- ------ -------------- ---------- -----
*.* root 215 rcm_daemon Unbound
*.111 daemon 567 rpcbind Idle
*.* daemon 567 rpcbind Unbound
*.55278 daemon 567 rpcbind Idle
*.* root 582 in.ndpd Unbound
TCP: IPv4
Local Address Remote Address User Pid Command Swind Send-Q Rwind Recv-Q State
-------------------- -------------------- -------- ------ -------------- ------ ------ ------ ------ -----------
*.22 *.* root 551 sshd 0 0 128000 0 LISTEN
172.27.10.9.22 172.27.10.79.48460 af 554 sshd 134664 0 128872 0 ESTABLISHED
172.27.10.9.22 172.27.10.79.48460 root 552 sshd 134664 0 128872 0 ESTABLISHED
*.111 *.* daemon 567 rpcbind 0 0 128000 0 LISTEN
*.* *.* daemon 567 rpcbind 0 0 128000 0 IDLE
*.111 *.* daemon 567 rpcbind 0 0 128000 0 LISTEN
*.* *.* daemon 567 rpcbind 0 0 128000 0 IDLE
*.48184 *.* root 584 inetd 0 0 128000 0 LISTEN
*.43966 *.* root 584 inetd 0 0 128000 0 LISTEN
TCP: IPv6
Local Address Remote Address User Pid Command Swind Send-Q Rwind Recv-Q State If
--------------------------------- --------------------------------- -------- ------ -------------- ------ ------ ------ ------ ----------- -----
*.22 *.* root 551 sshd 0 0 128000 0 LISTEN
*.111 *.* daemon 567 rpcbind 0 0 128000 0 LISTEN
*.* *.* daemon 567 rpcbind 0 0 128000 0 IDLE
*.43966 *.* root 584 inetd 0 0 128000 0 LISTEN
Active UNIX domain sockets
Address Type User Pid Command Local Address Remote Address
---------------- ---------- -------- ------ -------------- --------------------------------------- ---------------------------------------
ffffff06f63d5410 dgram root 582 in.ndpd /var/run/in.ndpd_mib
ffffff06f63d57d0 stream-ord root 582 in.ndpd /var/run/in.ndpd_ipadm
ffffff06f63d5b90 stream-ord af 554 sshd
ffffff06f48b3048 stream-ord root 552 sshd
ffffff06f48b3408 stream-ord af 554 sshd
ffffff06f48b3408 stream-ord root 552 sshd
ffffff06f480ab78 stream-ord root 339 hald /var/run/dbus/system_bus_socket
ffffff06f47e7030 stream-ord root 339 hald /var/run/hald/dbus-krtTyfGU6F
ffffff06f47e7030 stream-ord root 353 hald-addon-sto /var/run/hald/dbus-krtTyfGU6F
ffffff06f47e73f0 stream-ord root 339 hald /var/run/hald/dbus-krtTyfGU6F
ffffff06f47e73f0 stream-ord root 353 hald-addon-sto /var/run/hald/dbus-krtTyfGU6F
ffffff06f47e77b0 stream-ord root 584 inetd /var/run/.inetd.uds
ffffff06f4726028 stream-ord root 276 dbus-daemon /var/run/dbus/system_bus_socket
ffffff06f47267a8 stream-ord root 339 hald /var/run/hald/dbus-krtTyfGU6F
ffffff06f47363e0 stream-ord root 349 hald-addon-sto /var/run/hald/dbus-krtTyfGU6F
ffffff06f47367a0 stream-ord root 339 hald /var/run/hald/dbus-krtTyfGU6F
ffffff06f4736b60 stream-ord root 339 hald /var/run/hald/dbus-krtTyfGU6F
ffffff06f467e018 stream-ord root 346 hald-addon-acp /var/run/hald/dbus-krtTyfGU6F
ffffff06f467e018 stream-ord root 345 hald-addon-cpu /var/run/hald/dbus-krtTyfGU6F
ffffff06f467e3d8 stream-ord root 346 hald-addon-acp /var/run/hald/dbus-krtTyfGU6F
ffffff06f467e3d8 stream-ord root 345 hald-addon-cpu /var/run/hald/dbus-krtTyfGU6F
ffffff06f467e798 stream-ord root 339 hald /var/run/hald/dbus-krtTyfGU6F
ffffff06f467eb58 stream-ord root 343 hald-addon-net /var/run/hald/dbus-krtTyfGU6F
ffffff06f3500010 stream-ord root 339 hald /var/run/hald/dbus-VAF61JEarP
ffffff06f3500010 stream-ord root 342 hald-runner /var/run/hald/dbus-VAF61JEarP
ffffff06f35003d0 stream-ord root 339 hald /var/run/hald/dbus-VAF61JEarP
ffffff06f35003d0 stream-ord root 342 hald-runner /var/run/hald/dbus-VAF61JEarP
ffffff06f3500790 stream-ord root 339 hald /var/run/hald/dbus-krtTyfGU6F
ffffff06f3500b50 stream-ord root 276 dbus-daemon
ffffff06f269f008 stream-ord root 339 hald
ffffff06f269f008 stream-ord root 276 dbus-daemon
ffffff06f269f3c8 stream-ord root 339 hald /var/run/hald/dbus-VAF61JEarP
ffffff06f269f3c8 stream-ord root 276 dbus-daemon /var/run/hald/dbus-VAF61JEarP
ffffff06f269f788 stream-ord root 276 dbus-daemon /var/run/dbus/system_bus_socket
bloody% pfexec netstat -anuv
UDP: IPv4
Local Address Remote Address User Pid State Command
-------------------- -------------------- -------- ------ ---------- --------------
*.* root 215 Unbound /usr/lib/rcm/rcm_daemon
*.* root 215 Unbound /usr/lib/rcm/rcm_daemon
*.111 daemon 567 Idle /usr/sbin/rpcbind
*.* daemon 567 Unbound /usr/sbin/rpcbind
*.55278 daemon 567 Idle /usr/sbin/rpcbind
*.111 daemon 567 Idle /usr/sbin/rpcbind
*.* daemon 567 Unbound /usr/sbin/rpcbind
*.62085 daemon 567 Idle /usr/sbin/rpcbind
*.* root 582 Unbound /usr/lib/inet/in.ndpd
UDP: IPv6
Local Address Remote Address User Pid State If Command
--------------------------------- --------------------------------- -------- ------ ---------- ----- --------------
*.* root 215 Unbound /usr/lib/rcm/rcm_daemon
*.111 daemon 567 Idle /usr/sbin/rpcbind
*.* daemon 567 Unbound /usr/sbin/rpcbind
*.55278 daemon 567 Idle /usr/sbin/rpcbind
*.* root 582 Unbound /usr/lib/inet/in.ndpd
TCP: IPv4
Local/Remote Address Swind Snext Suna Rwind Rnext Rack Rto Mss State User Pid Command
-------------------- ------ -------- -------- ------ -------- -------- ----- ----- ----------- -------- ------ --------------
*.22
*.* 0 00000000 00000000 128000 00000000 00000000 1125 536 LISTEN root 551 /usr/sbin/sshd
172.27.10.9.22
172.27.10.79.48460 134664 810d599c 810d599c 128872 3429c4a2 3429c4a2 450 1448 ESTABLISHED af 554 /usr/sbin/sshd -R
172.27.10.9.22
172.27.10.79.48460 134664 810d599c 810d599c 128872 3429c4a2 3429c4a2 450 1448 ESTABLISHED root 552 /usr/sbin/sshd -R
*.111
*.* 0 00000000 00000000 128000 00000000 00000000 1125 1220 LISTEN daemon 567 /usr/sbin/rpcbind
*.*
*.* 0 00000000 00000000 128000 00000000 00000000 1125 1220 IDLE daemon 567 /usr/sbin/rpcbind
*.111
*.* 0 00000000 00000000 128000 00000000 00000000 1125 536 LISTEN daemon 567 /usr/sbin/rpcbind
*.*
*.* 0 00000000 00000000 128000 00000000 00000000 1125 536 IDLE daemon 567 /usr/sbin/rpcbind
*.48184
*.* 0 00000000 00000000 128000 00000000 00000000 1125 536 LISTEN root 584 /usr/lib/inet/inetd start
*.43966
*.* 0 00000000 00000000 128000 00000000 00000000 1125 1220 LISTEN root 584 /usr/lib/inet/inetd start
TCP: IPv6
Local/Remote Address Swind Snext Suna Rwind Rnext Rack Rto Mss State If User Pid Command
--------------------------------- ------ -------- -------- ------ -------- -------- ----- ----- ----------- ----- -------- ------ --------------
*.22
*.* 0 00000000 00000000 128000 00000000 00000000 1125 1220 LISTEN root 551 /usr/sbin/sshd
*.111
*.* 0 00000000 00000000 128000 00000000 00000000 1125 1220 LISTEN daemon 567 /usr/sbin/rpcbind
*.*
*.* 0 00000000 00000000 128000 00000000 00000000 1125 1220 IDLE daemon 567 /usr/sbin/rpcbind
*.43966
*.* 0 00000000 00000000 128000 00000000 00000000 1125 1220 LISTEN root 584 /usr/lib/inet/inetd start
Active UNIX domain sockets
Address Type User Pid Local Address Remote Address Command
---------------- ---------- -------- ------ --------------------------------------- --------------------------------------- --------------
ffffff06f63d5410 dgram root 582 /var/run/in.ndpd_mib /usr/lib/inet/in.ndpd
ffffff06f63d57d0 stream-ord root 582 /var/run/in.ndpd_ipadm /usr/lib/inet/in.ndpd
ffffff06f63d5b90 stream-ord af 554 /usr/sbin/sshd -R
ffffff06f48b3048 stream-ord root 552 /usr/sbin/sshd -R
ffffff06f48b3408 stream-ord af 554 /usr/sbin/sshd -R
ffffff06f48b3408 stream-ord root 552 /usr/sbin/sshd -R
ffffff06f480ab78 stream-ord root 339 /var/run/dbus/system_bus_socket /usr/lib/hal/hald --daemon=yes
ffffff06f47e7030 stream-ord root 339 /var/run/hald/dbus-krtTyfGU6F /usr/lib/hal/hald --daemon=yes
ffffff06f47e7030 stream-ord root 353 /var/run/hald/dbus-krtTyfGU6F /usr/lib/hal/hald-addon-storage
ffffff06f47e73f0 stream-ord root 339 /var/run/hald/dbus-krtTyfGU6F /usr/lib/hal/hald --daemon=yes
ffffff06f47e73f0 stream-ord root 353 /var/run/hald/dbus-krtTyfGU6F /usr/lib/hal/hald-addon-storage
ffffff06f47e77b0 stream-ord root 584 /var/run/.inetd.uds /usr/lib/inet/inetd start
ffffff06f4726028 stream-ord root 276 /var/run/dbus/system_bus_socket /usr/lib/dbus-daemon --system
ffffff06f47267a8 stream-ord root 339 /var/run/hald/dbus-krtTyfGU6F /usr/lib/hal/hald --daemon=yes
ffffff06f47363e0 stream-ord root 349 /var/run/hald/dbus-krtTyfGU6F /usr/lib/hal/hald-addon-storage
ffffff06f47367a0 stream-ord root 339 /var/run/hald/dbus-krtTyfGU6F /usr/lib/hal/hald --daemon=yes
ffffff06f4736b60 stream-ord root 339 /var/run/hald/dbus-krtTyfGU6F /usr/lib/hal/hald --daemon=yes
ffffff06f467e018 stream-ord root 346 /var/run/hald/dbus-krtTyfGU6F /usr/lib/hal/hald-addon-acpi
ffffff06f467e018 stream-ord root 345 /var/run/hald/dbus-krtTyfGU6F /usr/lib/hal/hald-addon-cpufreq
ffffff06f467e3d8 stream-ord root 346 /var/run/hald/dbus-krtTyfGU6F /usr/lib/hal/hald-addon-acpi
ffffff06f467e3d8 stream-ord root 345 /var/run/hald/dbus-krtTyfGU6F /usr/lib/hal/hald-addon-cpufreq
ffffff06f467e798 stream-ord root 339 /var/run/hald/dbus-krtTyfGU6F /usr/lib/hal/hald --daemon=yes
ffffff06f467eb58 stream-ord root 343 /var/run/hald/dbus-krtTyfGU6F /usr/lib/hal/hald-addon-network-discovery
ffffff06f3500010 stream-ord root 339 /var/run/hald/dbus-VAF61JEarP /usr/lib/hal/hald --daemon=yes
ffffff06f3500010 stream-ord root 342 /var/run/hald/dbus-VAF61JEarP hald-runner
ffffff06f35003d0 stream-ord root 339 /var/run/hald/dbus-VAF61JEarP /usr/lib/hal/hald --daemon=yes
ffffff06f35003d0 stream-ord root 342 /var/run/hald/dbus-VAF61JEarP hald-runner
ffffff06f3500790 stream-ord root 339 /var/run/hald/dbus-krtTyfGU6F /usr/lib/hal/hald --daemon=yes
ffffff06f3500b50 stream-ord root 276 /usr/lib/dbus-daemon --system
ffffff06f269f008 stream-ord root 339 /usr/lib/hal/hald --daemon=yes
ffffff06f269f008 stream-ord root 276 /usr/lib/dbus-daemon --system
ffffff06f269f3c8 stream-ord root 339 /var/run/hald/dbus-VAF61JEarP /usr/lib/hal/hald --daemon=yes
ffffff06f269f3c8 stream-ord root 276 /var/run/hald/dbus-VAF61JEarP /usr/lib/dbus-daemon --system
ffffff06f269f788 stream-ord root 276 /var/run/dbus/system_bus_socket /usr/lib/dbus-daemon --system
Updated by Andy Fiddaman over 3 years ago
This has been running in OmniOS for a while and has been tested in a variety of ways.
For testing that the output of netstat has not materially changed, apart from that column headings are not consistent across outputs, variants of the following script were used:
mkdir out
t()
{
op="out/`echo $* | tr ' ' '_' | sed 's/^-//'`"
netstat "$@" > $op.new
netstat~ "$@" > $op.old
diff -u $op.{old,new} > $op.diff && rm -f $op.diff
diff -uw $op.{old,new} > $op.diffw && rm -f $op.diffw
}
for f in inet inet6; do
for p in tcp udp; do
t -n -f $f -P $p
t -n -f $f -P $p -v
t -an -f $f -P $p
t -an -f $f -P $p -v
t -s -f $f -P $p
done
done
t -in
t -D
t -rn
t -m
t -M
Other tests have involved using netstat -u to check that information was present for various protocols and families and for both POSIX and TLI/XTI sockets. netstat -u has been run in a loop on a busy system and during system shutdown. Test programs have been written that create TLI sockets in various states (unbound, listening, etc.) and the output check against the expected results.
Updated by Electric Monk over 3 years ago
- Status changed from In Progress to Closed
- % Done changed from 90 to 100
git commit 78a2e113edb6fe0a0382b403b55d92e8f0bba78f
commit 78a2e113edb6fe0a0382b403b55d92e8f0bba78f
Author: Andy Fiddaman <omnios@citrus-it.co.uk>
Date: 2019-10-24T20:12:56.000Z
9531 Want netstat -u to show PIDs associated with sockets
11707 provide 64-bit libdhcpagent
11708 netstat should be smatch and CERRWARN clean
Portions contributed by: Mohamed A. Khalfella <khalfella@gmail.com>
Reviewed by: Robert Mustacchi <rm@fingolfin.org>
Reviewed by: Dominik Hassler <hadfl@omniosce.org>
Approved by: Garrett D'Amore <garrett@damore.org>
Actions