Project

General

Profile

Actions

Bug #13723

closed

expose bhyve metrics as kstats

Added by Patrick Mooney 4 months ago. Updated 3 months ago.

Status:
Closed
Priority:
Normal
Category:
bhyve
Start date:
Due date:
% Done:

100%

Estimated time:
Difficulty:
Medium
Tags:
Gerrit CR:

Description

The in-kernel components of bhyve measure certain aspects of VM operation such as vCPU time (idle vs running), interrupts, and events. It would be nice to expose via the kstat facility the portions of those measurements which are useful and likely to be somewhat stable.


Related issues

Related to illumos gate - Feature #13741: want microstates for bhyve vCPUsClosedPatrick Mooney

Actions
Actions #1

Updated by Patrick Mooney 3 months ago

  • Status changed from New to In Progress
Actions #2

Updated by Electric Monk 3 months ago

  • Gerrit CR set to 1431
Actions #3

Updated by Patrick Mooney 3 months ago

This (along with #13741) was tested on both Intel and AMD machines. Different guests were fired up to verify that vCPU microstate information was recorded and available via kstats:

# kstat -m vmm -n vcpu0
module: vmm                             instance: 1
name:   vcpu0                           class:    misc
        crtime                          261.817459864
        snaptime                        290.757305269
        time_emu_kern                   3101102297
        time_emu_user                   830303231
        time_idle                       10931261522
        time_init                       1683340745
        time_run                        12196115685
        time_sched                      196386817
        vcpu                            0

Metrics for all of the actually-running vCPUs reported non-zero as expected, while the non-running ones (from the highest specified vCPU to VM_MAXCPU) all reported zeros since they weren't active.

The proper VM-wide instance name record was also present:

module: vmm                             instance: 1
name:   vm                              class:    misc
        crtime                          261.817453817
        snaptime                        276.010007660
        vm_name                         centos0

The old running and idle tick metrics previously exposed via bhyvectl were confirmed to be absent.

Actions #4

Updated by Patrick Mooney 3 months ago

Actions #5

Updated by Patrick Mooney 3 months ago

Mike Zeller tested this with bhyve instances in zones to confirm kstat visibility:

From the gz I am now able to do:

# kstat -m vmm -n vm
module: vmm                             instance: 1
name:   vm                              class:    misc
crtime                          230.284875228
snaptime                        1414.128823131
vm_name                         SYSbhyve-37

module: vmm                             instance: 2
name:   vm                              class:    misc
crtime                          441.091924072
snaptime                        1414.128862559
vm_name                         SYSbhyve-38

And a sample of one instance of vcpu:

# kstat -m vmm -i 2
module: vmm                             instance: 2
name:   vcpu0                           class:    misc
crtime                          441.091926763
snaptime                        1003.042816315
time_emu_kern                   5096912634
time_emu_user                   7592775247
time_idle                       294039887535
time_init                       285462778
time_run                        254705869178
time_sched                      228361768
vcpu                            0

Actions #6

Updated by Patrick Mooney 3 months ago

Testing across reboots of a guest, I confirmed that active vCPUs maintained their time accounting across the reboot (values climbing upward, and not resetting to 0 at the reboot) and non-active vCPUs remained zeroed out:

root@carbide:~# kstat -m vmm -n vcpu0
module: vmm                             instance: 1
name:   vcpu0                           class:    misc
        crtime                          720.210690157
        snaptime                        1534.501657485
        time_emu_kern                   4186829339
        time_emu_user                   4267081104
        time_idle                       783414770165
        time_init                       1124367034
        time_run                        20342925172
        time_sched                      954389555
        vcpu                            0

root@carbide:~# kstat -m vmm -n vcpu5
module: vmm                             instance: 1
name:   vcpu5                           class:    misc
        crtime                          720.210704314
        snaptime                        1537.439880670
        time_emu_kern                   0
        time_emu_user                   0
        time_idle                       0
        time_init                       0
        time_run                        0
        time_sched                      0
        vcpu                            5
Actions #7

Updated by Electric Monk 3 months ago

  • Status changed from In Progress to Closed
  • % Done changed from 0 to 100

git commit 59460b493cd70b38e999010c370d57e58b989e45

commit  59460b493cd70b38e999010c370d57e58b989e45
Author: Patrick Mooney <pmooney@pfmooney.com>
Date:   2021-05-04T04:51:12.000Z

    13723 expose bhyve metrics as kstats
    13741 want microstates for bhyve vCPUs
    Reviewed by: Andy Fiddaman <andy@omnios.org>
    Reviewed by: Jason King <jason.brian.king@gmail.com>
    Reviewed by: Mike Zeller <mike.zeller@joyent.com>
    Approved by: Dan McDonald <danmcd@joyent.com>

Actions

Also available in: Atom PDF