Project

General

Profile

Actions

Bug #4866

open

arc_meta eats up all the ram and kills the system

Added by Dawid Stawiarski about 8 years ago. Updated about 7 years ago.

Status:
New
Priority:
Normal
Assignee:
-
Category:
-
Start date:
2014-05-15
Due date:
% Done:

0%

Estimated time:
Difficulty:
Medium
Tags:
needs-triage
Gerrit CR:

Description

description: arc_meta exceeds the limit shown (can't be set anymore and it's autotuned by system), eats ALL the ram and makes system unresponsive. almost the same issue was reported in ZFSonLinux: https://github.com/zfsonlinux/zfs/issues/676 - in our case the problem happens with >= ~25 rsyncs and/or the same number of simultanous finds. the arc_meta is not freed (well, actually it is IF the load is small enough, like ~10-15 rsyncs). to reproduce the problem we need about 30 minutes of rsync and/or find running (that's the time needed for arc_meta to use all the memory).

system info:
uname: SunOS system 5.11 oi_151a9 i86pc i386 i86pc Solaris
ram: 128GB
zpool size: ~60TB (32 mirrored disks + L2ARC ssd)

  1. cat /etc/system
    set zfs:zfs_arc_max = 68719476736
    set zfs:zfs_arc_meta_limit = 68719476736 ** actually don't work anymore
    set swapfs_minfree = 8387467
    set lotsfree = 1048433
    set zfs:zfs_arc_shrink_shift = 9

additional info:
1. we don't use deduplication nor compression
2. the problem also happens without L2ARC
3. the same problem happens on the latest OmniOS (omnios-8c08411)

data gathered, when system is still working:
  1. echo ::arc | mdb -k | grep "^arc_meta\\|^size\\|^p$\\|^c"
    c = 32768 MB
    c_min = 32768 MB
    c_max = 65536 MB
    size = 86564 MB
    arc_meta_used = 86564 MB
    arc_meta_limit = 65536 MB
    arc_meta_max = 86564 MB
  1. vmstat 2
    kthr memory page disk faults cpu
    r b w swap free re mf pi po fr de sr DT ro s2 s3 in sy cs us sy id
    1 0 0 8150832 2075552 645 630 16 1791 3222 2751512 10790571 4431 322 51 53 47612 27280 193023 0 11 89
    1 0 0 8152028 2078492 362 385 10 921 1660 2228728 9186209 4064 164 47 70 45099 23655 185690 0 9 91
    2 0 0 8152028 2077344 371 387 0 644 946 1805272 9681532 4449 270 58 59 55006 25465 191488 0 11 89
    0 0 0 8151900 2082676 728 750 2 1400 1822 1462276 7504854 4174 102 68 45 49385 29894 184519 0 9 91

as you can see:
1. arc_meta_used is much above limit (and arc size is above the max as well)
2. the swapfs_minfree don't help - the vmstat above show, that there's 2GB free memory but swapping starts happening - that's before arc_meta eats the rest and system dies (we need to power cycle the server)

Actions #1

Updated by Piotr Jasiukajtis about 8 years ago

  • Project changed from OpenIndiana Distribution to illumos gate
Actions #2

Updated by Dawid Stawiarski about 7 years ago

it looks like the problem weas related to configuration (it was copied from other machine running Nexenta) - after fresh start it didn't happen again.

so this issue can be considered false alarm.

Actions

Also available in: Atom PDF