Project

General

Profile

Bug #3331

deferred frees should happen after sync_pass 1

Added by Christopher Siden about 8 years ago. Updated about 8 years ago.

Status:
Closed
Priority:
Normal
Category:
zfs - Zettabyte File System
Start date:
2012-10-31
Due date:
% Done:

0%

Estimated time:
Difficulty:
Medium
Tags:
needs-triage
Gerrit CR:

Description

From George Wilson's original bug report at Delphix:

Today, we allow spa_sync() to process frees in both sync pass 1 and 2. Looking
at dcenter we can see that this is causing us to take longer to reach sync
convergence:

                pass 1      45350KB        326ms (65%) [avg size  78KB]
                pass 2       1171KB         29ms ( 5%) [avg size  26KB]
                pass 3        221KB         41ms ( 8%) [avg size   4KB]
             pass > 3        1941KB         82ms (16%) [avg size  21KB]
         metaslab_sync          0KB          1ms ( 0%) [avg size   0KB]
    metaslab_sync_done          0KB          2ms ( 0%) [avg size   0KB]
                 other          0KB          6ms ( 1%) [avg size   0KB]
      spa_free_sync_cb          0KB         10ms ( 2%) [avg size   0KB]
spa_sync(txg=2926592) took        498ms (7 passes)

                pass 1      46982KB        338ms (53%) [avg size  77KB]
                pass 2       1419KB         39ms ( 6%) [avg size  24KB]
                pass 3        244KB         35ms ( 5%) [avg size   3KB]
             pass > 3        1944KB        204ms (32%) [avg size  19KB]
         metaslab_sync          0KB          1ms ( 0%) [avg size   0KB]
    metaslab_sync_done          0KB          2ms ( 0%) [avg size   0KB]
                 other          0KB          5ms ( 0%) [avg size   0KB]
      spa_free_sync_cb          0KB          9ms ( 1%) [avg size   0KB]
spa_sync(txg=2926593) took        637ms (7 passes)

We can see that it's taking 7 sync passes to converge and we're writing about
2MB of data in passes > 3 (with an avg I/O size of 20KB).

Changing the SYNC_PASS_DEFERRED_FREE value from 2 to 1 we see the following:

                pass 1      43373KB        270ms (70%) [avg size  79KB]
                pass 2       1110KB         42ms (11%) [avg size  20KB]
                pass 3        156KB         24ms ( 6%) [avg size   4KB]
             pass > 3         156KB         29ms ( 7%) [avg size   4KB]
         metaslab_sync          0KB          1ms ( 0%) [avg size   0KB]
    metaslab_sync_done          0KB          2ms ( 0%) [avg size   0KB]
                 other          0KB          4ms ( 1%) [avg size   0KB]
      spa_free_sync_cb          0KB          8ms ( 2%) [avg size   0KB]
spa_sync(txg=2926617) took        383ms (4 passes)

                pass 1      46855KB        242ms (68%) [avg size  77KB]
                pass 2       1337KB         40ms (11%) [avg size  22KB]
                pass 3        161KB         23ms ( 6%) [avg size   4KB]
             pass > 3         161KB         25ms ( 7%) [avg size   4KB]
         metaslab_sync          0KB          1ms ( 0%) [avg size   0KB]
    metaslab_sync_done          0KB          2ms ( 0%) [avg size   0KB]
                 other          0KB          6ms ( 1%) [avg size   0KB]
      spa_free_sync_cb          0KB         11ms ( 3%) [avg size   0KB]
spa_sync(txg=2926618) took        352ms (4 passes)

Now it's taking 4 passes to converge and we're only writing out about 159KB
after sync pass 3 (avg I/O size of 4KB).

Related issues

Related to illumos gate - Feature #3335: make SYNC_PASS_* constants tunableClosedChristopher Siden2012-11-01

Actions
#1

Updated by Christopher Siden about 8 years ago

  • Status changed from In Progress to Closed
commit 01f55e48fb4d524eaf70687728aa51b7762e2e97
Author: George Wilson <george.wilson@delphix.com>
Date:   Wed Nov 7 14:05:17 2012 -0800

    3329 spa_sync() spends 10-20% of its time in spa_free_sync_cb()
    3330 space_seg_t should have its own kmem_cache
    3331 deferred frees should happen after sync_pass 1
    3335 make SYNC_PASS_* constants tunable
    Reviewed by: Adam Leventhal <ahl@delphix.com>
    Reviewed by: Matt Ahrens <matthew.ahrens@delphix.com>
    Reviewed by: Christopher Siden <chris.siden@delphix.com>
    Reviewed by: Eric Schrock <eric.schrock@delphix.com>
    Reviewed by: Richard Lowe <richlowe@richlowe.net>
    Reviewed by: Dan McDonald <danmcd@nexenta.com>
    Approved by: Eric Schrock <eric.schrock@delphix.com>

Also available in: Atom PDF