Bug #7362
ZFS ARC memory leak?
Status:
New
Priority:
Normal
Assignee:
-
Category:
-
Start date:
2016-09-07
Due date:
% Done:
0%
Estimated time:
Difficulty:
Medium
Tags:
needs-triage
Gerrit CR:
Description
One of our machines ran into a (relative) low memory situation.
ARC is capped to 768GB, arcstat shows this, arc_summary shows this, but ::memstat shows ~1.1TB used, confirmed by ::kmastat.
This is a SmartOS host running a single zone (with a single lofs mount for postgresql data folder, 640GB memory allocated).
I'm saying that is a relative low memory situation because we are monitoring pi/po/sr in vmstat and pi/po started to jump when database has increased workload (e.g. auto-vacuums).
Those missing ~300GB would've been handy for load spikes.
# echo ::memstat | mdb -k Page Summary Pages MB %Tot ------------ ---------------- ---------------- ---- Kernel 84790511 331212 21% Boot pages 68308 266 0% ZFS File Data 288887749 1128467 72% Anon 14527507 56748 4% Exec and libs 2916 11 0% Page cache 64381 251 0% Free (cachelist) 9707 37 0% Free (freelist) 14288100 55812 4% Total 402639179 1572809 Physical 402639175 1572809
# echo ::kmastat | mdb -k | grep ^z | grep G zfs_file_data_8192 8K 69441511 89594080 684G 242699750 0 zio_data_buf_8192 8K 23934432 69441511 530G 2646358325 0 zio_buf_16384 16K 2043669 2198952 33.6G 587133698 0 zio_buf_131072 128K 14301 40075 4.89G 552028122 0 zio_data_buf_131072 128K 4281590 4676590 571G 351784180 0
# arcstat time read miss miss% dmis dm% pmis pm% mmis mm% arcsz c 15:32:42 0 0 0 0 0 0 0 0 0 767G 768G
# arc_summary System Memory: Physical RAM: 1572809 MB Free Memory : 55922 MB LotsFree: 24575 MB ZFS Tunables (/etc/system): ARC Size: Current Size: 786431 MB (arcsize) Target Size (Adaptive): 786432 MB (c) Min Size (Hard Limit): 261120 MB (zfs_arc_min) Max Size (Hard Limit): 786432 MB (zfs_arc_max)
# kstat -p zfs:0:arcstats | grep arc_meta zfs:0:arcstats:arc_meta_limit 68719476736 zfs:0:arcstats:arc_meta_max 101617090664 zfs:0:arcstats:arc_meta_min 136902082560 zfs:0:arcstats:arc_meta_used 66633531112
# uname -a SunOS ec-f4-bb-ca-6c-b8 5.11 joyent_20160603T232338Z i86pc i386 i86pc
Updated by Alexandru Pirvulescu over 4 years ago
I've live re-tuned ARC from 768 to 500GB.
# arcstat time read miss miss% dmis dm% pmis pm% mmis mm% arcsz c 16:10:10 0 0 0 0 0 0 0 0 0 499G 500G
but ::memstat shows the same values (even less, losing about 10GB/day).
Page Summary Pages MB %Tot ------------ ---------------- ---------------- ---- Kernel 87021271 339926 22% Boot pages 68308 266 0% ZFS File Data 288887819 1128468 72% Anon 15627371 61044 4% Exec and libs 2928 11 0% Page cache 64466 251 0% Free (cachelist) 10978 42 0% Free (freelist) 10956038 42797 3%