Project

General

Profile

Bug #6569

large file delete can starve out write ops

Added by Alek Pinchuk almost 4 years ago. Updated almost 3 years ago.

Status:
Closed
Priority:
Normal
Assignee:
Category:
zfs - Zettabyte File System
Start date:
2016-01-20
Due date:
% Done:

100%

Estimated time:
Difficulty:
Medium
Tags:

Description

The core issue I've found is that there is no throttle for how many deletes get assigned to one TXG. As a results when deleting large files we end up filling consecutive TXGs with deletes/frees, then write throttling other (more important) ops.

There is an easy test case for this problem. Try deleting several large files (at least 1/2 TB) while you do write ops on the same pool. What we've seen is performance of these write ops (let's call it sideload I/O) would drop to zero.

More specifically the problem is that dmu_free_long_range_impl() can/will fill up all of the dirty data in the pool "instantly", before many of the sideload ops can get in. So sideload performance will be impacted until all the files are freed.
The solution we have tested at Nexenta (with positive results) creates a relatively simple throttle for how many "free" ops we let into one TXG.

However this solution exposes other problems that should also be addressed. If we are to slow down freeing of data that means one has to wait even longer (assuming vnode ref count of 1) to get shell back after an rm or for NFS thread to finish the free-ing op.
To avoid this the proposed solution is to call zfs_inactive() async for "large" files. Async freeing then begs for the reclaimed space to be accounted for in the zpool's "freeing" prop.
The other issue with having a longer delete is the inability to export/unmount for a longer period of time. The proposed solution is to interrupt freeing of blocks when a fs is unmounted.


Related issues

Related to illumos gate - Bug #5911: ZFS "hangs" while deleting fileClosed2015-05-04

Actions
Related to illumos gate - Bug #7301: zpool export -f should be able to interrupt file freeingClosed2016-08-13

Actions

History

#1

Updated by Alek Pinchuk almost 4 years ago

  • Related to Bug #5911: ZFS "hangs" while deleting file added
#2

Updated by Alek Pinchuk over 3 years ago

  • Related to Bug #7301: zpool export -f should be able to interrupt file freeing added
#3

Updated by Electric Monk almost 3 years ago

  • Status changed from New to Closed
  • % Done changed from 80 to 100

git commit ff5177ee8bf9a355131ce2cc61ae2da6a5a6fdd6

commit  ff5177ee8bf9a355131ce2cc61ae2da6a5a6fdd6
Author: Alek Pinchuk <alek@nexenta.com>
Date:   2016-12-31T02:06:43.000Z

    6569 large file delete can starve out write ops
    Reviewed by: Matt Ahrens <mahrens@delphix.com>
    Reviewed by: Sanjay Nadkarni <sanjay.nadkarni@nexenta.com>
    Reviewed by: Pavel Zakharov <pavel.zakharov@delphix.com>
    Approved by: Dan McDonald <danmcd@omniti.com>

Also available in: Atom PDF