Project

General

Profile

Bug #4754

io issued to near-full luns even after setting noalloc threshold

Added by Christopher Siden over 5 years ago. Updated over 5 years ago.

Status:
Closed
Priority:
Normal
Category:
zfs - Zettabyte File System
Start date:
2014-04-14
Due date:
% Done:

100%

Estimated time:
Difficulty:
Medium
Tags:
needs-triage

Description

Original bug:

zfs_mg_noalloc_threshold is set to 15. After setting that, 10sec sample for
zpool iostat is:

-------------------
    capacity     operations    bandwidth
pool        alloc   free   read  write   read  write
----------  -----  -----  -----  -----  -----  -----
domain0     3.35T  1.12T  5.21K  1.74K  93.6M  98.1M
  c2t1d0     202G  95.8G    370    163  6.59M  6.41M
  c2t2d0     265G  33.0G    548     58  8.83M  4.62M
  c2t3d0     196G   102G    443    159  8.28M  6.90M
  c2t4d0     268G  30.3G    418     58  8.15M  4.69M
  c2t5d0     266G  31.7G    415     58  8.20M  4.62M
  c2t6d0     265G  32.7G    419     58  8.13M  4.77M
  c2t8d0     265G  32.7G    491     58  8.66M  4.59M
  c2t9d0     266G  32.4G    411     58  8.14M  4.77M
  c2t10d0    266G  32.0G    412     58  8.29M  4.67M
  c2t11d0    267G  30.7G    492     58  8.86M  4.62M
  c2t12d0    236G   162G    237    334  3.05M  12.0M
  c2t13d0    233G   165G    230    189  2.91M  11.9M
  c2t14d0    227G   171G    239    258  3.08M  11.8M
  c2t15d0    205G   193G    208    212  2.42M  11.8M
----------------------

Notice that there is still significant amount of IO done to the 30G some free
luns. George asked me to run couple dtrace scripts:

1) 
metaslab_group_allocatable:entry
/stringof(args[0]->mg_vd->vdev_spa->spa_name) == "domain0"/
{
        self->vd = args[0]->mg_vd;
}
metaslab_group_allocatable:return
/self->vd/
{
        @[stringof(self->vd->vdev_path), args[1]] = count();
        self->vd = 0;
}

30sec sample o/p:

  /dev/dsk/c2t2d0s0                                         0                5
  /dev/dsk/c2t3d0s0                                         1               14
  /dev/dsk/c2t4d0s0                                         0               14
  /dev/dsk/c2t5d0s0                                         0               15
  /dev/dsk/c2t6d0s0                                         0               22
  /dev/dsk/c2t8d0s0                                         0               23
  /dev/dsk/c2t9d0s0                                         0               29
  /dev/dsk/c2t10d0s0                                        0               35
  /dev/dsk/c2t11d0s0                                        0               43
  /dev/dsk/c2t1d0s0                                         1               63
  /dev/dsk/c2t15d0s0                                        1              324
  /dev/dsk/c2t12d0s0                                        1              427
  /dev/dsk/c2t13d0s0                                        1              869
  /dev/dsk/c2t14d0s0                                        1              920

2) /home/gwilson/dstat/zil/metaslab_group_alloc_by_vdev.d

Have run that script and will update the bug once I get the data from the
customer's box.

Just wanted to point that this is a SQL box and hence all the IO is through
zvol.

Analysis by George Wilson:

With the introduction of zfs_mg_noalloc_threshold we can now avoid allocations
to devices that have a free capacity below zfs_mg_noalloc_threshold. However,
synchronous writes don't utilize this logic because they are not allowed to
"fast gang". As a result all synchronous writes will still try to allocate on
full devices.  Sync write allocations that are not going to a slog should use
zfs_mg_noalloc_threshold functionality. This can be accomplished by removing
the METASLAB_GANG_AVOID flag in zio_alloc_zil().

History

#1

Updated by Christopher Siden over 5 years ago

  • Status changed from New to Closed
commit b6240e830b871f59c22a3918aebb3b36c872edba
Author: George Wilson <george.wilson@delphix.com>
Date:   Fri Apr 18 09:35:03 2014

    4754 io issued to near-full luns even after setting noalloc threshold
    4755 mg_alloc_failures is no longer needed
    Reviewed by: Matthew Ahrens <mahrens@delphix.com>
    Reviewed by: Adam Leventhal <ahl@delphix.com>
    Reviewed by: Dan McDonald <danmcd@omniti.com>
    Approved by: Dan McDonald <danmcd@omniti.com>

Also available in: Atom PDF