Project

General

Profile

Bug #175

zfs vdev cache consumes excessive memory

Added by Gordon Ross about 9 years ago. Updated almost 8 years ago.

Status:
Resolved
Priority:
Normal
Category:
kernel
Start date:
2010-09-08
Due date:
% Done:

0%

Estimated time:
Difficulty:
Medium
Tags:
needs-triage

Description

See OpenSolaris CR 6684116

but note that the work-around suggested there is less than ideal.
Better to set zfs_vdev_cache_size to zero, i.e. in /etc/system
set zfs:zfs_vdev_cache_size = 0

After some performance verification, it might make sense to
change the default to zero in uts/common/fs/zfs/vdev_cache.c

History

#1

Updated by Garrett D'Amore over 8 years ago

  • Category set to kernel
  • Assignee set to Garrett D'Amore

Solaris 11 has been running with the tunable set to zero for a while now. Eventually we ought to just remove the code.

#2

Updated by Garrett D'Amore over 8 years ago

We end up storing 10MB of readahead per vdev, and so if you have many
vdevs (say, if you sell storage systems), that really sucks on main
memory.

We find that the cache is usually very under utilized in the field. (30%-70%)

#3

Updated by Garrett D'Amore over 8 years ago

  • Status changed from New to Resolved

Resolved in:

garrett@thinkpad{15}> hg export tip
  1. HG changeset patch
  2. User Garrett D'Amore <>
  3. Date 1303458581 25200
  4. Node ID f3ce1af7c12d88b7e9513aad2e7e1b20cae5bab0
  5. Parent 20c193a013b852a0435f528a95759f9684d9115a
    175 zfs vdev cache consumes excessive memory
    Reviewed by: George Wilson <>
    Reviewed by: Eric Schrock <>
    Approved by: Richard Lowe <>
#4

Updated by Jim Klimov almost 8 years ago

  • Difficulty set to Medium
  • Tags set to needs-triage

I object to removal of the code for everybody based on its inefficiency for somebody. For example, vdev prefetch was very efficient during a scrub on my systems and boosted the scrub speed (minus several hours off the total count for two scrubs).
In fact, I propose to expand the feature by making a separate rolling cache for non-metadata sectors which are currently discarded from prefetched data.
Tuning the two cache sizes is the end-user's adventure, and zero-size defaults can protect the multiple-drive systems with miniature RAM. illumos's task is to provide the mechanisms that can help it suit everyone - from single-spindle laptops to petabyte arrays ;)
Details are tracked here: https://www.illumos.org/issues/2017

Also available in: Atom PDF