illumos gate

With the introduction of an illumos-based KVM looming, it would be nice to prepare for it by pulling in Joyent's implementation kvmstat. It's a simple kstat reader that shows useful per-second output across all KVM virtual machines. For example, here's one second of output on a machine running two VMs (one 2 VCPU instance running Linux; another 4 VCPU instance running the illumos-derived SmartOS):

   pid vcpu |  exits :  haltx   irqx  irqwx    iox  mmiox |   irqs   emul   eptv
  4668    0 |     23 :      6      0      0      1      0 |      6     16      0
  4668    1 |     25 :      6      1      0      1      0 |      6     16      0
  5026    0 |  17833 :    223   2946    707    106      0 |   3379  13315      0
  5026    1 |  18687 :    244   2761    512      0      0 |   3085  14803      0
  5026    2 |  15696 :    194   3452    542      0      0 |   3568  11230      0
  5026    3 |  16822 :    244   2817    487      0      0 |   3100  12963      0

As for the meaning of the columns, they are explained with "kvmstat -h":

# kvmstat -h
Usage: kvmstat [interval [count]]

  Displays statistics for running kernel virtual machines, with one line
  per virtual CPU.  All statistics are reported as per-second rates.

  The columns are as follows:

    pid    =>  identifier of process controlling the virtual CPU
    vcpu   =>  virtual CPU identifier relative to its virtual machine
    exits  =>  virtual machine exits for the virtual CPU
    haltx  =>  virtual machine exits due to the HLT instruction
    irqx   =>  virtual machine exits due to a pending external interrupt
    irqwx  =>  virtual machine exits due to an open interrupt window
    iox    =>  virtual machine exits due to an I/O instruction
    mmiox  =>  virtual machine exits due to memory mapped I/O 
    irqs   =>  interrupts injected into the virtual CPU
    emul   =>  instructions emulated in the kernel
    eptv   =>  extended page table violations

On machines that do not have the (unbundled) kvm driver, kvmstat will (like most *stat tools) simply display its headers (without any output) once per second.