Bug #954


ZFS metadata uses way too much space with 4Kb blocks (ashift=12)

Added by Jim Klimov over 10 years ago. Updated almost 10 years ago.

zfs - Zettabyte File System
Start date:
Due date:
% Done:


Estimated time:
Gerrit CR:


Details, from problem question to my research and findings, are represented in OpenSolaris forum thread:

In short, due to modern drives with native 4Kb sectors being slower when used with smaller sectors (i.e. emulated 512-byte ones), my pool was created with a modified zpool binary which enforced ashift=12 and 4Kb blocks became the minimum for ZFS.

Now it appears that filesystem overhead grew considerably, especially with regard to volumes with a small volblocksize (in my extreme case every data block of 4Kb volume is matched by a 4Kb metadata block, at least doubling the space usage in the pool vs. "user data size"), and with regard to snapshots (they seem to reserve space for doubling the metadata blocks - which is quite a lot now).

One proposal would be to raise the top blocksize from 128Kb to 1Mb so as to keep the FS metadata's overhead percentage smaller.

Another proposal is to treat such 4Kb metadata blocks as containers with eight colocated 512-byte blocks, each with a different piece of ZFS metadata, i.e. by adding or expanding a layer of abstraction to address the metadata blocks.

This idea seems to be backwards-compatible in terms of code which might expect metadata blocks to be (at least) 512 bytes in size, and would keep the overhead as low as with standard ashift=9 pools, and should do little harm to performance on 4Kb-sectored hard disks (or SSD media).

Related issues

Related to illumos gate - Feature #1044: ZFS: Allow specifying minimum dataset/metaslab block size AND alignmentNew2011-05-17

Actions #1

Updated by Albert Lee over 10 years ago

  • Project changed from OpenIndiana Distribution to illumos gate
Actions #2

Updated by Jim Klimov about 10 years ago

  • Difficulty set to Medium
  • Tags set to needs-triage

Today I've thought of an alternative solution which might be better compatible with ZFS on-disk format: instead of reworking ZFS metadata processing for several entries to be "packaged" in one ZFS 4kb-block, we could formally use 512-byte blocks for the ZFS but larger blocks (4kb or more - but with guaranteed start-alignment).

I'll detail that alternative idea in another bug, and I like it better already - the change would be in ZFS-write logic but not producing an incompatible disk format.

Posted as bug #1044.

Actions #3

Updated by Albert Lee almost 10 years ago

  • Category set to zfs - Zettabyte File System
  • Difficulty changed from Medium to Expert
  • Tags changed from needs-triage to zfs

Also available in: Atom PDF