Fix ARC hit rate
https://github.com/zfsonlinux/zfs/pull/6989 needs to be merged. Since dbuf cache size increased recently, it becomes more important for buffers promotion MRU to MFU.
When the compressed ARC feature was added in commit d3c2ae1 the method of reference counting in the ARC was modified. As part of this accounting change the arc_buf_add_ref() function was removed entirely.
This would have been fine but the arc_buf_add_ref() function served a second undocumented purpose of updating the ARC access information when taking a hold on a dbuf. Without this logic in place a cached dbuf would not migrate its associated arc_buf_hdr_t to the MFU list. This negatively impacts the ARC hit rate, particularly on systems with a small ARC.
This change reinstates the missing call to arc_access() from dbuf_hold() by implementing a new arc_buf_access() function.
Motivation and Context¶
As reported in ZOL issue 6171 a significant drop in the ARC hit rate was reported by users upgrading to the 0.7.x releases.
How Has This Been Tested? (in the original ZOL pull request, not on illumos)¶
Manually with the following trivial test case which reads a file twice. After the first read the file size should be reflected in the arcstat mru_size. After the second read the file size should have been migrated to the arcstat mfu_size.
Notice that when testing without the patch the mfu_size doesn't increase on the second read.
$ zfs mount -a $ grep -E 'mru_size|mfu_size' /proc/spl/kstat/zfs/arcstats mru_size 4 412672 mfu_size 4 739328 $ cat /tank/file >/dev/null grep -E 'mru_size|mfu_size' /proc/spl/kstat/zfs/arcstats mru_size 4 21483008 mfu_size 4 786432 $ cat /tank/file >/dev/null grep -E 'mru_size|mfu_size' /proc/spl/kstat/zfs/arcstats mru_size 4 21483008 mfu_size 4 786432
After applying this patch the behavior is once again as expected. This matches the long standing behavior from the zfs-0.6.5.x releases.
$ zfs mount -a $ grep -E 'mru_size|mfu_size' /proc/spl/kstat/zfs/arcstats mru_size 4 415232 mfu_size 4 737792 $ cat /tank/file >/dev/null $ grep -E 'mru_size|mfu_size' /proc/spl/kstat/zfs/arcstats mru_size 4 21384192 mfu_size 4 889856 $ cat /tank/file >/dev/null $ grep -E 'mru_size|mfu_size' /proc/spl/kstat/zfs/arcstats mru_size 4 1084928 mfu_size 4 21198336
Additional testing included a full local run of the ZTS.
Updated by Joshua Clulow 4 months ago
On the subject of performance:
We are testing with recordsize=8k, either cached reads or reads from very fast SSD's. 100,000 IOPS or higher.
We saw ~8% regression when doing a 100% cached read case from the dbuf cache. The regression is not as severe with larger blocksizes (3% for 128K).
Updated by Electric Monk 4 months ago
- Status changed from New to Closed
- % Done changed from 0 to 100
commit 7b38fab6300e0a1599b9741cfeafd94d362b87aa Author: Alexander Motin <mav@FreeBSD.org> Date: 2019-03-08T17:19:26.000Z 9433 Fix ARC hit rate Reviewed by: Giuseppe Di Natale <firstname.lastname@example.org> Reviewed by: Tony Hutter <email@example.com> Reviewed by: Tim Chase <firstname.lastname@example.org> Reviewed by: George Wilson <email@example.com> Reviewed by: George Melikov <firstname.lastname@example.org> Reviewed by: Brian Behlendorf <email@example.com> Reviewed by: Matt Ahrens <firstname.lastname@example.org> Reviewed by: Serapheim Dimitropoulos <email@example.com> Reviewed by: George Wilson <firstname.lastname@example.org> Reviewed by: Brad Lewis <email@example.com> Approved by: Dan McDonald <firstname.lastname@example.org>