Project

General

Profile

Bug #1909

disk sync write perf regression when slog is used post oi_148

Added by Frederik Wessels over 7 years ago. Updated over 7 years ago.

Status:
Resolved
Priority:
Normal
Assignee:
Category:
zfs - Zettabyte File System
Start date:
2011-12-20
Due date:
% Done:

70%

Estimated time:
Difficulty:
Medium
Tags:
needs-triage

Description

Initially I discovered this while testing VMWare ESXi 4.1u2 on latest Illumos bits. I found out that the write performance was almost halved.
My initial guess was that TCP/IP or NFS was the culprit. But various tests including an iperf version for ESXi showed that wirespeed could be achieved. A second hardware platform, AMD instead of XEON, showed the same regression. Apart from other similarities both platforms were using a dedicated ddr ram based device connected via onboard sata. And as soon as I removed the slog from the zpool sync performance was the same between oi148,oi151a and illumos nightly. Albeit somewhat low 5MB/s, but that could be expected from a two disk pool. As soon as the slog was added back to the pool oi148 sync writes jumped to 50MB/s but illumos only increased to 13MB/s. Showing a strange pattern in zilstat. The async performance remained the same on all versions with or without slog.

Here's a summary from attached logs:
  1. dd if=/dev/zero of=/mpool/4g.bin [oflag=sync] bs=32K count=128K
    This more or less simulates the esxi nfs writes sync and async. I tried various block sizes but that didn’t made much of a difference
    For example a 2 disk pool with and without slog
    oi148 with slog async dd=218MB/s
    oi148 with slog sync dd=50MB/s zilstat iops/per txg=7500
    oi148 NO slog async dd=216MB/s
    oi148 NO slog sync dd=5MB/s zilstat iops/per txg=750

illumos with slog async dd=223MB/s
illumos with slog sync dd=13MB/s zilstat iops/per txg= first=4400 second=675 third=750 this pattern repeats
illumos NO slog async dd=221MB/s
illumos NO slog sync dd=5MB/s zilstat iops/per txg=770

Here's a bit of data regarding the slog device standalone
illumos spool async dd=148MB/s
illumos spool sync dd=53.7MB/s (the device is using itself for zil)
Not lightning fast but with low latency, no wear, and sufficient for gbit ethernet

The regression is introduced after oi148 but before oi151a. At least that limits the set of commits to analyze.
More data can be provided on request.


Files

dd_32KB_4GB_oi148.txt (19 KB) dd_32KB_4GB_oi148.txt perf data DD on oi148 WITH slog Frederik Wessels, 2011-12-20 07:18 PM
dd_32KB_4GB_oi148_without_slog.txt (36.8 KB) dd_32KB_4GB_oi148_without_slog.txt perf data DD on oi148 withOUT slog Frederik Wessels, 2011-12-20 07:18 PM
dd_32KB_4GB_illumos.txt (19.6 KB) dd_32KB_4GB_illumos.txt perf data DD on illumos WITH slog Frederik Wessels, 2011-12-20 07:18 PM
dd_32KB_4GB_illumos_without_slog.txt (13.8 KB) dd_32KB_4GB_illumos_without_slog.txt perf data DD on illumos withOUT slog Frederik Wessels, 2011-12-20 07:18 PM

History

#1

Updated by George Wilson over 7 years ago

  • Status changed from New to In Progress
  • % Done changed from 0 to 70

The issue is that ZIL blocks should always be contiguous blocks and thus the "fast gang" logic should not apply to those allocations. I have generated a patch and am currently running some tests.

A simple workaround is to do the following:

  1. echo "zfs_mg_alloc_failures/W 0t10000" | mdb -kw
#2

Updated by Albert Lee over 7 years ago

  • Assignee set to George Wilson
#3

Updated by Eric Schrock over 7 years ago

  • Status changed from In Progress to Resolved

changeset: 13572:85c66b89d5f2
tag: tip
user: George Wilson <>
date: Mon Jan 23 18:47:28 2012 -0800

description:
1909 disk sync write perf regression when slog is used post oi_148
Reviewed by: Matt Ahrens <>
Reviewed by: Eric Schrock <>
Reviewed by: Robert Mustacchi <>
Reviewed by: Bill Pijewski <>
Reviewed by: Richard Elling <>
Reviewed by: Steve Gonczi <>
Reviewed by: Garrett D'Amore <>
Reviewed by: Dan McDonald <>
Reviewed by: Albert Lee <>
Approved by: Eric Schrock <>

modified:
usr/src/uts/common/fs/zfs/metaslab.c
usr/src/uts/common/fs/zfs/zio.c

Also available in: Atom PDF