Project

General

Profile

Actions

Bug #13389

closed

After persistent L2ARC import, cache device has constant 8KB/sec load

Added by Adam Stylinski 9 months ago. Updated 7 months ago.

Status:
Closed
Priority:
Normal
Assignee:
Category:
zfs - Zettabyte File System
Start date:
Due date:
% Done:

100%

Estimated time:
Difficulty:
Medium
Tags:
Gerrit CR:

Description

I can't say for certain which revision caused this, but I'm seeing a constant 8KB/sec load on my L2ARC device in my pool, regardless of how idle it is.

                             capacity     operations     bandwidth 
pool                       alloc   free   read  write   read  write
-------------------------  -----  -----  -----  -----  -----  -----
rpool1                      107G   115G      0      0      0      0
  c8t5001B44F00970028d0s0   107G   115G      0      0      0      0
-------------------------  -----  -----  -----  -----  -----  -----
sqlpool                    29.8G   192G      0      0      0      0
  mirror                   9.98G  64.0G      0      0      0      0
    c8t50014EE6034B2CC7d0      -      -      0      0      0      0
    c8t50014EE0ABA3B6B9d0      -      -      0      0      0      0
  mirror                   9.97G  64.0G      0      0      0      0
    c5t4d0                     -      -      0      0      0      0
    c5t5d0                     -      -      0      0      0      0
  mirror                   9.88G  64.1G      0      0      0      0
    c5t1d0                     -      -      0      0      0      0
    c8t50014EE057470E4Bd0      -      -      0      0      0      0
logs                           -      -      -      -      -      -
  c15tE4D25CFBDE0F0100d0     40K  54.5G      0      0      0      0
cache                          -      -      -      -      -      -
  c5t2d0                   4.98G  88.1G      0      0      0  7.95K
-------------------------  -----  -----  -----  -----  -----  -----
                             capacity     operations     bandwidth 
pool                       alloc   free   read  write   read  write
-------------------------  -----  -----  -----  -----  -----  -----
rpool1                      107G   115G      0      0      0      0
  c8t5001B44F00970028d0s0   107G   115G      0      0      0      0
-------------------------  -----  -----  -----  -----  -----  -----
sqlpool                    29.8G   192G      0      0      0      0
  mirror                   9.98G  64.0G      0      0      0      0
    c8t50014EE6034B2CC7d0      -      -      0      0      0      0
    c8t50014EE0ABA3B6B9d0      -      -      0      0      0      0
  mirror                   9.97G  64.0G      0      0      0      0
    c5t4d0                     -      -      0      0      0      0
    c5t5d0                     -      -      0      0      0      0
  mirror                   9.88G  64.1G      0      0      0      0
    c5t1d0                     -      -      0      0      0      0
    c8t50014EE057470E4Bd0      -      -      0      0      0      0
logs                           -      -      -      -      -      -
  c15tE4D25CFBDE0F0100d0     40K  54.5G      0      0      0      0
cache                          -      -      -      -      -      -
  c5t2d0                   4.98G  88.1G      0      0      0  7.96K
-------------------------  -----  -----  -----  -----  -----  -----
                    extended device statistics              
    r/s    w/s   kr/s   kw/s wait actv wsvc_t asvc_t  %w  %b device
    0.0    1.6    0.0  113.3  0.0  0.0    0.0    0.2   0   0 c15tE4D25CFBDE0F0100d0
    0.1    0.2    2.5    3.9  0.0  0.0    1.5    0.2   0   0 rpool1
    0.0    1.1    0.1   45.2  0.0  0.0    0.0    0.3   0   0 c5t2d0
    0.1    0.9    3.4   28.3  0.0  0.0    0.2    0.9   0   0 c5t4d0
    0.1    0.9    3.6   28.3  0.0  0.0    0.1    0.9   0   0 c5t5d0
    0.1    0.9    3.7   28.2  0.0  0.0    0.1    0.4   0   0 c5t1d0
    0.1    0.9    3.6   28.2  0.0  0.0    0.0    1.0   0   0 c8t50014EE057470E4Bd0
    0.1    0.9    3.4   28.2  0.0  0.0    0.0    1.0   0   0 c8t50014EE0ABA3B6B9d0
    0.1    0.9    3.9   28.2  0.0  0.0    0.0    0.4   0   0 c8t50014EE6034B2CC7d0
    0.1    0.2    2.5    3.9  0.0  0.0    0.0    0.2   0   0 c8t5001B44F00970028d0
    0.6    8.1   21.8  327.8  0.1  0.0   11.1    0.7   0   0 sqlpool
                    extended device statistics              
    r/s    w/s   kr/s   kw/s wait actv wsvc_t asvc_t  %w  %b device
    0.0    1.0    0.0    8.0  0.0  0.0    0.0    0.2   0   0 c5t2d0
    0.0    1.0    0.0    8.0  0.0  0.0    0.0    0.3   0   0 sqlpool
                    extended device statistics              
    r/s    w/s   kr/s   kw/s wait actv wsvc_t asvc_t  %w  %b device
    0.0    1.0    0.0    8.0  0.0  0.0    0.0    0.2   0   0 c5t2d0
    0.0    1.0    0.0    8.0  0.0  0.0    0.0    0.2   0   0 sqlpool
                    extended device statistics              
    r/s    w/s   kr/s   kw/s wait actv wsvc_t asvc_t  %w  %b device
    0.0    1.0    0.0    8.0  0.0  0.0    0.0    0.2   0   0 c5t2d0
    0.0    1.0    0.0    8.0  0.0  0.0    0.0    0.2   0   0 sqlpool
                    extended device statistics              
    r/s    w/s   kr/s   kw/s wait actv wsvc_t asvc_t  %w  %b device
    0.0    1.0    0.0    8.0  0.0  0.0    0.0    0.2   0   0 c5t2d0
    0.0    1.0    0.0    8.0  0.0  0.0    0.0    0.3   0   0 sqlpool
                    extended device statistics              
    r/s    w/s   kr/s   kw/s wait actv wsvc_t asvc_t  %w  %b device
    0.0    1.0    0.0    8.0  0.0  0.0    0.0    0.2   0   0 c5t2d0
    0.0    1.0    0.0    8.0  0.0  0.0    0.0    0.3   0   0 sqlpool

For the sake of not slowly killing the flash, it'd be nice to get to the bottom of this at some point (even if it is cheap TLC).


Files

persist_l2arc_008_pos.txt (7.54 KB) persist_l2arc_008_pos.txt stdout of passing test Jason King, 2021-02-22 07:13 PM
Actions #1

Updated by Adam Stylinski 9 months ago

Also should add, this is what's in /etc/system:
set zfs:zfs_arc_max = 1073741824
set zfs:zfs_prefetch_disable = 1

Actions #3

Updated by Adam Stylinski 8 months ago

Yes, that seems like a safe bet. Can someone maybe merge that PR into Illumos-gate?

Actions #4

Updated by Dan McDonald 8 months ago

Analysis from OpenZFS is here: https://github.com/openzfs/zfs/pull/11537

Actions #5

Updated by Dan McDonald 8 months ago

Bug filer has confirmed no more 8kb/sec writes into L2ARC.

Running zfstests now to make sure there's no regression. Once that's confirmed, I'll RTI. This is one of the two test notes.

Actions #6

Updated by Dan McDonald 8 months ago

  • Category set to zfs - Zettabyte File System
  • Assignee set to Dan McDonald
Actions #7

Updated by Adam Stylinski 7 months ago

Dan mentioned it already but for posterity's sake - the patch removes the constant 8KB/sec write workload on the L2ARC for me.

Actions #8

Updated by Dan McDonald 7 months ago

ZFS tests showed no regressions save one: persist_l2arc_008_pos.

Turns out that with this bug fixed, my lightweight test environment didn't generate the load needed to pass. Luckily for me Jason King found a rig that could: https://gist.github.com/jasonbking/97ac660daae43b4ce2bf58ce32e477ee

Actions #9

Updated by Jason King 7 months ago

The persist_l2arc_008_pos was failing on Dan's test system. Looking at the failure, as well as conferring with George Amankis (who wrote the tests, as well as did a lot of recent work w/ the L2ARC), we believe the failure is that the test does not always generate enough I/O to cause data to be pushed out to the ARC. When it doesn't, the test currently fails.

When I re-ran the persistent L2ARC tests on the test VM I use, they all passed -- including persist_l2arc_008_pos. Since that one had failed, I also looked at the stdout from the test, and it all looks reasonable and as expected as best as I can tell.

Actions #10

Updated by Jason King 7 months ago

Just for reference, I've also attached the same stdout passing output in case it's useful in the future.

Actions #11

Updated by Electric Monk 7 months ago

  • Status changed from New to Closed
  • % Done changed from 0 to 100

git commit b639505692797add0a13ee545fd6ca20d63f89fd

commit  b639505692797add0a13ee545fd6ca20d63f89fd
Author: George Amanakis <gamanakis@gmail.com>
Date:   2021-02-23T00:42:23.000Z

    13389 After persistent L2ARC import, cache device has constant 8KB/sec load
    Reviewed by: Brian Behlendorf <behlendorf1@llnl.gov>
    Reviewed by: Dan McDonald <danmcd@joyent.com>
    Reviewed by: Toomas Soome <tsoome@me.com>
    Reviewed by: Jason King <jbk@joyent.com>
    Approved by: Gordon Ross <gordon.w.ross@gmail.com>

Actions

Also available in: Atom PDF