illumos gate

One of our machines ran into a (relative) low memory situation.
ARC is capped to 768GB, arcstat shows this, arc_summary shows this, but ::memstat shows ~1.1TB used, confirmed by ::kmastat.

This is a SmartOS host running a single zone (with a single lofs mount for postgresql data folder, 640GB memory allocated).
I'm saying that is a relative low memory situation because we are monitoring pi/po/sr in vmstat and pi/po started to jump when database has increased workload (e.g. auto-vacuums).

Those missing ~300GB would've been handy for load spikes.

# echo ::memstat | mdb -k
Page Summary                Pages                MB  %Tot
------------     ----------------  ----------------  ----
Kernel                   84790511            331212   21%
Boot pages                  68308               266    0%
ZFS File Data           288887749           1128467   72%
Anon                     14527507             56748    4%
Exec and libs                2916                11    0%
Page cache                  64381               251    0%
Free (cachelist)             9707                37    0%
Free (freelist)          14288100             55812    4%

Total                   402639179           1572809
Physical                402639175           1572809

# echo ::kmastat | mdb -k | grep ^z | grep G
zfs_file_data_8192                8K  69441511  89594080   684G  242699750     0
zio_data_buf_8192                 8K  23934432  69441511   530G 2646358325     0
zio_buf_16384                    16K   2043669   2198952  33.6G  587133698     0
zio_buf_131072                  128K     14301     40075  4.89G  552028122     0
zio_data_buf_131072             128K   4281590   4676590   571G  351784180     0

# arcstat
    time  read  miss  miss%  dmis  dm%  pmis  pm%  mmis  mm%  arcsz     c
15:32:42     0     0      0     0    0     0    0     0    0   767G  768G

# arc_summary
System Memory:
         Physical RAM:  1572809 MB
         Free Memory :  55922 MB
         LotsFree:      24575 MB

ZFS Tunables (/etc/system):

ARC Size:
         Current Size:             786431 MB (arcsize)
         Target Size (Adaptive):   786432 MB (c)
         Min Size (Hard Limit):    261120 MB (zfs_arc_min)
         Max Size (Hard Limit):    786432 MB (zfs_arc_max)

# kstat -p zfs:0:arcstats | grep arc_meta
zfs:0:arcstats:arc_meta_limit   68719476736
zfs:0:arcstats:arc_meta_max     101617090664
zfs:0:arcstats:arc_meta_min     136902082560
zfs:0:arcstats:arc_meta_used    66633531112

# uname -a
SunOS ec-f4-bb-ca-6c-b8 5.11 joyent_20160603T232338Z i86pc i386 i86pc