Project

General

Profile

Bug #10052

"dladm show-ether" should pick one kstat snapshot and stick with it

Added by Joshua M. Clulow 9 months ago. Updated 6 months ago.

Status:
Closed
Priority:
Normal
Assignee:
-
Category:
networking
Start date:
2018-12-08
Due date:
% Done:

100%

Estimated time:
Difficulty:
Medium
Tags:

Description

When dladm show-ether is invoked, especially on more recent systems, there is a long pause before information about each NIC is emitted. As it turns out, the command opens and closes the kstat device ~56 times per NIC.

System with one NIC:

$ pfexec time dtrace -c 'dladm show-ether' -n 'syscall::open:entry /pid == $target/ { self->x = arg0; } syscall::open:return /arg0 >= 0 & self->x != NULL/ { @[copyinstr(self->x)] = count(); } syscall::open:return /self->x != NULL/ { self->x = NULL; }'
dtrace: description 'syscall::open:entry ' matched 3 probes
LINK            PTYPE    STATE    AUTO  SPEED-DUPLEX                    PAUSE
vioif0          current  up       no    1G-f                            none
dtrace: pid 1969 has exited

  /dev/dld                                                          1
  /etc/svc/volatile/dladm/dlmgmt_door                               1
  /usr/lib/locale//en_US.UTF-8/LC_COLLATE/LCL_DATA                  1
  /usr/lib/locale//en_US.UTF-8/LC_CTYPE/LCL_DATA                    1
  /usr/lib/locale//en_US.UTF-8/LC_MESSAGES/LCL_DATA                 1
  /usr/lib/locale//en_US.UTF-8/LC_MONETARY/LCL_DATA                 1
  /usr/lib/locale//en_US.UTF-8/LC_NUMERIC/LCL_DATA                  1
  /usr/lib/locale//en_US.UTF-8/LC_TIME/LCL_DATA                     1
  /dev/kstat                                                       56

real        2.8
user        1.8
sys         0.9

System with four NICs:

[root@headnode (lab) ~]# time dtrace -c 'dladm show-ether' -n 'syscall::open:entry /pid == $target/ { self->x = arg0; } syscall::open:return /arg0 >= 0 & self->x != NULL/ { @[copyinstr(self->x)] = count(); } syscall::open:return /self->x != NULL/ { self->x = NULL; }'
dtrace: description 'syscall::open:entry ' matched 3 probes
LINK            PTYPE    STATE    AUTO  SPEED-DUPLEX                    PAUSE
vioif0          current  up       no    1G-f                            none
vioif1          current  up       no    1G-f                            none
vioif2          current  up       no    1G-f                            none
vioif3          current  up       no    1G-f                            none
dtrace: pid 50415 has exited

  /dev/dld                                                          1
  /etc/svc/volatile/dladm/dlmgmt_door                               1
  /usr/lib/locale//en_US.UTF-8/LC_COLLATE/LCL_DATA                  1
  /usr/lib/locale//en_US.UTF-8/LC_CTYPE/LCL_DATA                    1
  /usr/lib/locale//en_US.UTF-8/LC_MESSAGES/LCL_DATA                 1
  /usr/lib/locale//en_US.UTF-8/LC_MONETARY/LCL_DATA                 1
  /usr/lib/locale//en_US.UTF-8/LC_NUMERIC/LCL_DATA                  1
  /usr/lib/locale//en_US.UTF-8/LC_TIME/LCL_DATA                     1
  /dev/kstat                                                      224

real    0m4.714s
user    0m1.086s
sys     0m3.572s

There are several routines in libdladm which kstat_open(), read a single kstat, then kstat_close(); e.g., dladm_get_single_mac_stat(). We should lazily open the kstat handle when required and cache it on the dladm_handle_t, rather than open and close it for each statistic.

History

#1

Updated by Carlos Neira 7 months ago

Hi,

This patch avoids repeated calls to kstat_open() in dladm by retaining a kstat_ctl_t pointer in dladm_handle_t, which is then used by kstat consumers.
The libkstat handle is opened when the first caller needs it. This allows us the library to share the kstat handle.

webrev : http://cr.illumos.org/~webrev/cneira/10052/

source files modified for this change.

:100644 100644 757fcb5149 edfc844a94 M usr/src/cmd/flowstat/flowstat.c
:100644 100644 b8e744f4f9 81d3fb2990 M usr/src/lib/libdladm/common/libdladm.c
:100644 100644 c2fceb25ab 36092f89a4 M usr/src/lib/libdladm/common/libdladm.h
:100644 100644 13169285e3 20db1cb1d7 M usr/src/lib/libdladm/common/libdladm_impl.h
:100644 100644 a4f9307bd7 40de09402b M usr/src/lib/libdladm/common/libdlstat.c
:100644 100644 c799723a91 deb9279f4a M usr/src/lib/libdladm/common/libdlstat.h

After changes:

pfexec time dtrace -c 'dladm show-ether' -n 'syscall::open:entry /pid == $target/ { self->x = arg0; } syscall::open:return /arg0 >= 0 & self->x != NULL/ { @[copyinstr(self->x)] = count(); } syscall::open:return /self->x != NULL/ { self->x = NULL; }'

dtrace: description 'syscall::open:entry ' matched 3 probes
LINK            PTYPE    STATE    AUTO  SPEED-DUPLEX                    PAUSE
e1000g0         current  up       yes   1G-f                            bi
dtrace: pid 109209 has exited
  /dev/dld                                                          1
  /dev/kstat                                                        1
  /etc/svc/volatile/dladm/dlmgmt_door                               1
  /usr/lib/locale//en_US.UTF-8/LC_COLLATE/LCL_DATA                  1
  /usr/lib/locale//en_US.UTF-8/LC_CTYPE/LCL_DATA                    1
  /usr/lib/locale//en_US.UTF-8/LC_MESSAGES/LCL_DATA                 1
  /usr/lib/locale//en_US.UTF-8/LC_MONETARY/LCL_DATA                 1
  /usr/lib/locale//en_US.UTF-8/LC_NUMERIC/LCL_DATA                  1
  /usr/lib/locale//en_US.UTF-8/LC_TIME/LCL_DATA                     1
real        0.4
user        0.6
sys         0.7
cneira@Trixie:…os/usr/src/lib/libdladm/common$ dlstat -i 10
           LINK    IPKTS   RBYTES    OPKTS   OBYTES
        e1000g0  108.83K   10.32M  157.52K   18.15M
          vnic1    1.56K   84.66K        0        0
          vnic0    1.56K   84.66K        0        0
          vnic2    1.56K   84.66K        0        0
           net6    1.56K   84.66K        0        0
        e1000g0       49    3.19K       90   10.25K
          vnic1        0        0        0        0
          vnic0        0        0        0        0
          vnic2        0        0        0        0
           net6        0        0        0        0
        e1000g0       47    3.16K       85    9.81K
          vnic1        0        0        0        0
          vnic0        0        0        0        0
          vnic2        0        0        0        0
           net6        0        0        0        0
        e1000g0       51    3.31K       92    9.99K
          vnic1        2       84        0        0
          vnic0        2       84        0        0
          vnic2        2       84        0        0
           net6        2       84        0        0
        e1000g0       51    3.31K       91   10.23K
          vnic1        0        0        0        0
          vnic0        0        0        0        0
          vnic2        0        0        0        0
           net6        0        0        0        0
neira@Trixie:…os/usr/src/lib/libdladm/common$ pfexec flowadm add-flow -l e1000g0 -a transport=tcp,local_port=22 sshd
cneira@Trixie:~$ flowstat -i 1
           FLOW    IPKTS   RBYTES    IERRS    OPKTS   OBYTES    OERRS
          smtp0        0        0        0        0        0        0
           sshd   15.44K    1.11M        0   22.44K    2.57M        0
          smtp0        0        0        0        0        0        0
           sshd        8      564        0       11    1.28K        0
          smtp0        0        0        0        0        0        0
           sshd        5      300        0        8      960        0
          smtp0        0        0        0        0        0        0
           sshd        5      300        0        9      998        0
          smtp0        0        0        0        0        0        0
           sshd        8      564        0       10    1.12K        0
          smtp0        0        0        0        0        0        0
           sshd        4      240        0        7      906        0
          smtp0        0        0        0        0        0        0
           sshd        5      300        0        8    1.01K        0
          smtp0        0        0        0        0        0        0
           sshd        7      504        0        9    1.03K        0
          smtp0        0        0        0        0        0        0
           sshd        5      300        0       10    1.18K        0
          smtp0        0        0        0        0        0        0
           sshd        5      300        0        8      944        0
cneira@Trixie:~$

Joshua Clulow wrote:

When dladm show-ether is invoked, especially on more recent systems, there is a long pause before information about each NIC is emitted. As it turns out, the command opens and closes the kstat device ~56 times per NIC.

System with one NIC:

[...]

System with four NICs:

[...]

There are several routines in libdladm which kstat_open(), read a single kstat, then kstat_close(); e.g., dladm_get_single_mac_stat(). We should lazily open the kstat handle when required and cache it on the dladm_handle_t, rather than open and close it for each statistic.

#2

Updated by Electric Monk 6 months ago

  • Status changed from New to Closed
  • % Done changed from 0 to 100

git commit 399dcf08ed1dc7e6abf8d7e5dc9447e11047c9f2

commit  399dcf08ed1dc7e6abf8d7e5dc9447e11047c9f2
Author: carlos antonio neira bustos <cneirabustos@gmail.com>
Date:   2019-02-11T08:12:27.000Z

    10052 "dladm show-ether" should pick one kstat snapshot and stick with it
    Reviewed by: Rob Johnston <rob.johnston@joyent.com>
    Reviewed by: Robert Mustacchi <rm@joyent.com>
    Reviewed by: Gergő Mihály Doma <domag02@gmail.com>
    Reviewed by: Andy Fiddaman <andy@omniosce.org>
    Approved by: Joshua M. Clulow <josh@sysmgr.org>

Also available in: Atom PDF