Project

General

Profile

Bug #9729

kstat not exposing non-global zone stat counters

Added by Alasdair Lumsden 11 months ago. Updated 5 months ago.

Status:
New
Priority:
Normal
Assignee:
-
Category:
kernel
Start date:
2018-08-13
Due date:
% Done:

0%

Estimated time:
Difficulty:
Medium
Tags:
needs-triage

Description

For a lot of modules, kstat in the global zone exposes both the gz stats and the ngz stats. For example:

  1. kstat -p tcp:*:tcp:inAckBytes
    tcp:0:tcp:inAckBytes 2034097154
    tcp:1:tcp:inAckBytes 466042423
    tcp:2:tcp:inAckBytes 1964265266
    tcp:18:tcp:inAckBytes 3669

However the udp module only exposes gz stats:

  1. kstat -p udp:*:udp:inDatagrams
    udp:0:udp:inDatagrams 902624

If we zlogin to a zone and run the same command, we do however get the per-zone kstat:

  1. zlogin 00398784-8bb1-414e-b38d-e5a38927c4ce kstat -p udp:*:udp:inDatagrams
    udp:0:udp:inDatagrams 8458274

This is making stats monitoring rather difficult, as we wish to gather all zone stats via the global zone, without the use of zlogin (which intrudes into customer environments).

Modules that appear to be broken in this way:

ip
udp
icmp
ilb
ipf
ipnet
sctp

History

#1

Updated by Alasdair Lumsden 11 months ago

I have found this commit by Bill Pijewski which updates illumos to expose the per-zone tcp kstats into the global zone, which looks like a basis for perhaps doing the same in other modules:

4303 want local zone TCP kstats in the global zone
https://github.com/illumos/illumos-gate/commit/4a0bd0711d2aa8c9386d9c275964fd49a79f7ea5

Hopefully this is helpful.

#2

Updated by Alasdair Lumsden 11 months ago

Following the model of the patch in 4303, I've prepared: https://gist.github.com/alaslums/807185d21a964a3824e0abf74c2ea810

This fixes up the following modules:

icmp
ip
ip6
sctp
udp

ilb and ipf can have multiple instances per netstack, so the indexing by stackid falls down. ipnet doesn't use kstat_create_netstack so I've left it alone.

I don't currently have a facility for building illumos-gate but will see about setting one up so I can test the changes.

#3

Updated by Alasdair Lumsden 11 months ago

I've now done this with ipfilter:

https://gist.github.com/alaslums/f609af779f723607e0e20388437964ff

To avoid repeated calls to net_getnetstackidbynetid, I've switched from net_kstat_create to kstat_create_netstack (which is used in most places anyway). I'm not sure if this is the right approach as net_kstat_create is a documented interface whereas kstat_create_netstack isn't.

I'm also not sure about the "error handling" approach of wrapping things in a big if block, but this approach was already used in this function (and presumably you'll never hit the case where a stackid can't be found, and if we do then this kstat doesn't get initialised which isn't the end of the world).

#4

Updated by Carlos Neira 5 months ago

Hi Alasdair,

I took your changes and tested them in Omniosce they are working for me. I just submit them for review.

https://illumos.org/rb/r/1474/

With your changes stats are exposed to GZ :

cneira@Trixie:/build/illumos-omnios$ zoneadm list -icv
  ID NAME             STATUS     PATH                           BRAND    IP
   0 global           running    /                              ipkg     shared
   1 t0               running    /zcage/vms/t0                  lx       excl
   - 9e19b8ed-ad9c-4284-a948-3cf803aaa3e0 configured /zcage/vms/9e19b8ed-ad9c-4284-a948-3cf803aaa3e0 lx       excl
   - 1fd17810-1517-47ad-8389-3557b74bdb11 configured /zcage/vms/1fd17810-1517-47ad-8389-3557b74bdb11 lx       excl
   - lxvm0            configured /zcage/vms/lxvm0               lx       excl
   - lxvm             configured /zcage/vms/lxvm                lx       excl
   - lxvm2            configured /zcage/vms/lxvm2               lx       excl
   - l1               configured /zcage/vms/l1                  lx       excl
cneira@Trixie:/build/illumos-omnios$ kstat -p ip:*:icmp:outMsgs
ip:0:icmp:outMsgs       0
ip:1:icmp:outMsgs       0
cneira@Trixie:/build/illumos-omnios$ kstat -p udp:*:udp:inDatagrams
udp:0:udp:inDatagrams   33
udp:1:udp:inDatagrams   0
cneira@Trixie:/build/illumos-omnios$ kstat -p tcp:*:tcp:inAckBytes
tcp:0:tcp:inAckBytes    321216
tcp:1:tcp:inAckBytes    0
omnios$ kstat -p sctp:*:sctp:sctpRtoMax
sctp:0:sctp:sctpRtoMax  60000
sctp:1:sctp:sctpRtoMax  60000

Bests

Alasdair Lumsden wrote:

I've now done this with ipfilter:

https://gist.github.com/alaslums/f609af779f723607e0e20388437964ff

To avoid repeated calls to net_getnetstackidbynetid, I've switched from net_kstat_create to kstat_create_netstack (which is used in most places anyway). I'm not sure if this is the right approach as net_kstat_create is a documented interface whereas kstat_create_netstack isn't.

I'm also not sure about the "error handling" approach of wrapping things in a big if block, but this approach was already used in this function (and presumably you'll never hit the case where a stackid can't be found, and if we do then this kstat doesn't get initialised which isn't the end of the world).

Also available in: Atom PDF