Project

General

Profile

Bug #9433

Fix ARC hit rate

Added by Alexander Motin over 1 year ago. Updated 4 months ago.

Status:
Closed
Priority:
Normal
Category:
zfs - Zettabyte File System
Start date:
2018-04-02
Due date:
% Done:

100%

Estimated time:
Difficulty:
Medium
Tags:

Description

https://github.com/zfsonlinux/zfs/pull/6989 needs to be merged. Since dbuf cache size increased recently, it becomes more important for buffers promotion MRU to MFU.

Description

When the compressed ARC feature was added in commit d3c2ae1 the method of reference counting in the ARC was modified. As part of this accounting change the arc_buf_add_ref() function was removed entirely.

This would have been fine but the arc_buf_add_ref() function served a second undocumented purpose of updating the ARC access information when taking a hold on a dbuf. Without this logic in place a cached dbuf would not migrate its associated arc_buf_hdr_t to the MFU list. This negatively impacts the ARC hit rate, particularly on systems with a small ARC.

This change reinstates the missing call to arc_access() from dbuf_hold() by implementing a new arc_buf_access() function.

Motivation and Context

As reported in ZOL issue 6171 a significant drop in the ARC hit rate was reported by users upgrading to the 0.7.x releases.

How Has This Been Tested? (in the original ZOL pull request, not on illumos)

Manually with the following trivial test case which reads a file twice. After the first read the file size should be reflected in the arcstat mru_size. After the second read the file size should have been migrated to the arcstat mfu_size.

Notice that when testing without the patch the mfu_size doesn't increase on the second read.

$ zfs mount -a
$ grep -E 'mru_size|mfu_size' /proc/spl/kstat/zfs/arcstats
mru_size                        4    412672
mfu_size                        4    739328

$ cat /tank/file >/dev/null
 grep -E 'mru_size|mfu_size' /proc/spl/kstat/zfs/arcstats
mru_size                        4    21483008
mfu_size                        4    786432

$ cat /tank/file >/dev/null
 grep -E 'mru_size|mfu_size' /proc/spl/kstat/zfs/arcstats
mru_size                        4    21483008
mfu_size                        4    786432

After applying this patch the behavior is once again as expected. This matches the long standing behavior from the zfs-0.6.5.x releases.

$ zfs mount -a
$ grep -E 'mru_size|mfu_size' /proc/spl/kstat/zfs/arcstats
mru_size                        4    415232
mfu_size                        4    737792

$  cat /tank/file >/dev/null
$ grep -E 'mru_size|mfu_size' /proc/spl/kstat/zfs/arcstats
mru_size                        4    21384192
mfu_size                        4    889856

$ cat /tank/file >/dev/null
$ grep -E 'mru_size|mfu_size' /proc/spl/kstat/zfs/arcstats
mru_size                        4    1084928
mfu_size                        4    21198336

Additional testing included a full local run of the ZTS.

History

#1

Updated by Joshua Clulow 4 months ago

  • Description updated (diff)
  • Tags deleted (needs-triage)
#2

Updated by Joshua Clulow 4 months ago

On the subject of performance:

Matthew Ahrens:

We are testing with recordsize=8k, either cached reads or reads from very fast SSD's. 100,000 IOPS or higher.

George Wilson:

We saw ~8% regression when doing a 100% cached read case from the dbuf cache. The regression is not as severe with larger blocksizes (3% for 128K).

#3

Updated by Electric Monk 4 months ago

  • Status changed from New to Closed
  • % Done changed from 0 to 100

git commit 7b38fab6300e0a1599b9741cfeafd94d362b87aa

commit  7b38fab6300e0a1599b9741cfeafd94d362b87aa
Author: Alexander Motin <mav@FreeBSD.org>
Date:   2019-03-08T17:19:26.000Z

    9433 Fix ARC hit rate
    Reviewed by: Giuseppe Di Natale <dinatale2@llnl.gov>
    Reviewed by: Tony Hutter <hutter2@llnl.gov>
    Reviewed by: Tim Chase <tim@chase2k.com>
    Reviewed by: George Wilson <george.wilson@delphix.com>
    Reviewed by: George Melikov <mail@gmelikov.ru>
    Reviewed by: Brian Behlendorf <behlendorf1@llnl.gov>
    Reviewed by: Matt Ahrens <matt@delphix.com>
    Reviewed by: Serapheim Dimitropoulos <serapheim.dimitro@delphix.com>
    Reviewed by: George Wilson <george.wilson@delphix.com>
    Reviewed by: Brad Lewis <brad.lewis@delphix.com>
    Approved by: Dan McDonald <danmcd@joyent.com>

Also available in: Atom PDF