poor performance of 1MB writes on certain RAID-Z configurations
RAID-Z requires that space be allocated in multiples of P+1 sectors,
because this is the minimum size block that can have the required amount
of parity. Thus blocks on RAIDZ1 must be allocated in a multiple of 2
sectors; on RAIDZ2 multiple of 3; and on RAIDZ3 multiple of 4. A sector
is a unit of 2^ashift bytes, typically 512B or 4KB.
To satisfy this constraint, the allocation size is rounded up to the
proper multiple, resulting in up to 3 "pad sectors" at the end of some
blocks. The contents of these pad sectors are not used, so we do not
need to read or write these sectors. However, some storage hardware
performs much worse (around 1/2 as fast) on mostly-contiguous writes
when there are small gaps of non-overwritten data between the writes.
Therefore, ZFS creates "optional" zio's when writing RAID-Z blocks that
include pad sectors. If writing a pad sector will fill the gap between
two (required) writes, we will issue the optional zio, thus doubling
performance. The gap-filling performance improvement was introduced in
Writing the optional zio is done by the io aggregation code in
vdev_queue.c. The problem is that it is also subject to the limit on
the size of aggregate writes, zfs_vdev_aggregation_limit, which is by
default 128KB. For a given block, if the amount of data plus padding
written to a leaf device exceeds zfs_vdev_aggregation_limit, the
optional zio will not be written, resulting in a ~2x performance
The problem occurs only for certain values of ashift, compressed block
size, and RAID-Z configuration (number of parity and data disks). It
cannot occur with the default recordsize=128KB. If compression is
enabled, all configurations with recordsize=1MB or larger will be
impacted to some degree.
The problem notably occurs with recordsize=1MB, compression=off, with 10
disks in a RAIDZ2 or RAIDZ3 group (with 512B or 4KB sectors). Therefore
this problem has been known as "the 1MB 10-wide RAIDZ2 (or 3) problem".
The problem also occurs with the following configurations:With recordsize=512KB or 256KB, compression=off, the problem occurs only
in rarely-used configurations:
- 4-wide RAIDZ1 with recordsize=512KB and ashift=12 (4KB sectors)
- 4-wide RAIDZ2 (either recordsize, either ashift)
- 5-wide RAIDZ2 with recordsize=512KB (either ashift)
- 6-wide RAIDZ2 with recordsize=512KB (either ashift)
- RAIDZ1 with 4 or 8 disks
- RAIDZ2 with 4, 8, or 10 disks
- RAIDZ3 with 6, 8, 9, or 10 disks
- RAIDZ1 with 7 or 8 disks
- RAIDZ2 with 4, 5, or 10 disks
- RAIDZ3 with 6, 9, or 10 disks
With recordsize=2MB and larger (which can only be selected by changing
kernel tunables), many configurations are affected, including with
higher numbers of disks (up to 18 disks with recordsize=2MB).
Increase zfs_vdev_aggregation_limit to allow the optional zio to be
aggregated, thus eliminating the problem. Setting it to 256KB fixes all
The solution is to aggregate optional zio's regardless of the
aggregation size limit.
Analysis sponsored by Intel Corp.
Updated by Electric Monk about 5 years ago
- Status changed from New to Closed
- % Done changed from 0 to 100
commit 5b062782532a1d5961c4a4b655906e1238c7c908 Author: Matthew Ahrens <firstname.lastname@example.org> Date: 2017-06-06T05:13:57.000Z 8005 poor performance of 1MB writes on certain RAID-Z configurations Reviewed by: Saso Kiselkov <email@example.com> Reviewed by: George Wilson <firstname.lastname@example.org> Reviewed by: Pavel Zakharov <email@example.com> Approved by: Robert Mustacchi <firstname.lastname@example.org>