Project

General

Profile

Bug #9486

reduce memory used by device removal on fragmented pools

Added by Matthew Ahrens over 1 year ago. Updated about 1 year ago.

Status:
Closed
Priority:
Normal
Category:
zfs - Zettabyte File System
Start date:
2018-04-16
Due date:
% Done:

100%

Estimated time:
Difficulty:
Medium
Tags:
needs-triage

Description

Device removal allocates a new location for each allocated segment on
the disk that's being removed. Each allocation results in one entry in
the mapping table, which maps from old location + length to new
location. When a fragmented disk is removed, this can result in a large
number of mapping entries, and thus a large amount of memory consumed by
the mapping table. In the worst real-world cases, we've seen around 1GB
of RAM per 1TB of storage removed.

We can improve on this situation by allocating larger segments, which
span across both allocated and free regions of the device being removed.
By including free regions in the allocation (and thus mapping), we
reduce the number of mapping entries. For example, if we have a 4K
allocation followed by 1K free and then 4K allocated, we would allocate
4+1+4 = 9KB, and then move the entire region (including allocated and
free parts). In this case we used one mapping where previously we would
have used two, but often the ratio is much higher (up to 20:1 in
real-world use). We then need to mark the regions that were free on the
removing device as free in the new locations, and also obsolete in the
mapping entry.

This method preserves the fragmentation of the removing device, rather
than consolidating its allocated space into a small number of chunks
where possible. But it results in drastic reduction of memory used by
the mapping table - around 20x in the most-fragmented cases.

In the most fragmented real-world cases, this reduces memory used by the
mapping from ~1GB to ~50MB of RAM per 1TB of storage removed. Less
fragmented cases will typically also see around 50-100MB of RAM per 1TB
of storage.

History

#1

Updated by Electric Monk about 1 year ago

  • % Done changed from 0 to 100
  • Status changed from New to Closed

git commit cfd63e1b1bcf7ba4bf72f55ddbd87ce008d2986d

commit  cfd63e1b1bcf7ba4bf72f55ddbd87ce008d2986d
Author: Matthew Ahrens <mahrens@delphix.com>
Date:   2018-06-01T16:22:51.000Z

    9486 reduce memory used by device removal on fragmented pools
    Reviewed by: George Wilson <george.wilson@delphix.com>
    Reviewed by: Serapheim Dimitropoulos <serapheim.dimitro@delphix.com>
    Reviewed by: Brian Behlendorf <behlendorf1@llnl.gov>
    Reviewed by: Tim Chase <tim@chase2k.com>
    Approved by: Robert Mustacchi <rm@joyent.com>

Also available in: Atom PDF