Project

General

Profile

Bug #7504

kmem_reap hangs spa_sync and administrative tasks

Added by Prakash Surya almost 3 years ago. Updated almost 3 years ago.

Status:
Closed
Priority:
Normal
Assignee:
Category:
zfs - Zettabyte File System
Start date:
2016-10-24
Due date:
% Done:

100%

Estimated time:
Difficulty:
Medium
Tags:
needs-triage

Description

We see long spa_sync(). We are waiting to hold dp_config_rwlock for writer. Some other thread holds dp_config_rwlock for reader, then calls arc_get_data_buf(), which finds that arc_is_overflowing()==B_TRUE. So it waits (while holding dp_config_rwlock for reader) for arc_reclaim_thread to signal arc_reclaim_waiters_cv. Before signaling, arc_reclaim_thread does arc_kmem_reap_now(), which takes ~seconds.

2016 Apr 22 04:54:16 held config lock for 2007ms from:
              zfs`dsl_pool_rele+0x23
              zfs`dmu_send_impl+0x5d4
              zfs`dmu_send+0x65d
              zfs`zfs_ioc_send_new+0x1ab
              zfs`zfsdev_ioctl+0x1ac
              genunix`cdev_ioctl+0x39
              specfs`spec_ioctl+0x60
              genunix`fop_ioctl+0x55
              genunix`ioctl+0x9b
              unix`sys_syscall+0x17a

slept for 1989ms from:
              genunix`cv_wait+0x70
              zfs`arc_get_data_buf+0x1d7
              zfs`arc_hdr_alloc_pdata+0x26
              zfs`arc_hdr_alloc+0xef
              zfs`arc_read+0x1bf
              zfs`dbuf_read_impl+0x17a
              zfs`dbuf_read+0xeb
              zfs`dmu_buf_hold+0x78
              zfs`zap_lockdir+0x6d
              zfs`zap_lookup_norm+0x60
              zfs`zap_lookup+0x46
              zfs`dsl_dir_hold+0x111
              zfs`dsl_dataset_hold+0x49
              zfs`dmu_send+0x169
              zfs`zfs_ioc_send_new+0x1ab
              zfs`zfsdev_ioctl+0x1ac
              genunix`cdev_ioctl+0x39
              specfs`spec_ioctl+0x60
              genunix`fop_ioctl+0x55
              genunix`ioctl+0x9b

spent 1528ms reaping zio_data_buf_6144, freed 574MB, (destroyed 49033 slabs of 12KB each)
spent 527ms reaping zio_data_buf_81920, freed 951MB, (destroyed 12180 slabs of 80KB each)
  2016 Apr 22 04:54:16  spent 2337ms in arc_kmem_reap_now
<pre>

History

#1

Updated by Electric Monk almost 3 years ago

  • % Done changed from 0 to 100
  • Status changed from New to Closed

git commit 405a5a0f5c3ab36cb76559467d1a62ba648bd809

commit  405a5a0f5c3ab36cb76559467d1a62ba648bd809
Author: Matthew Ahrens <mahrens@delphix.com>
Date:   2016-11-07T16:54:54.000Z

    7504 kmem_reap hangs spa_sync and administrative tasks
    Reviewed by: George Wilson <george.wilson@delphix.com>
    Reviewed by: Prakash Surya <prakash.surya@delphix.com>
    Approved by: Dan McDonald <danmcd@omniti.com>

Also available in: Atom PDF