Project

General

Profile

Bug #8021

ARC buf data scatter-ization

Added by Daniel Kimmel over 2 years ago. Updated over 2 years ago.

Status:
Closed
Priority:
Normal
Assignee:
Category:
zfs - Zettabyte File System
Start date:
2017-03-31
Due date:
% Done:

100%

Estimated time:
Difficulty:
Medium
Tags:

Description

The ARC buf data project (known simply as "ABD" since its genesis in the ZoL community) changes the way the ARC allocates `b_pdata` memory from using linear `void *` buffers to using scatter/gather lists of fixed-size 1KB chunks. This improves ZFS's performance by helping to defragment the address space occupied by the ARC, in particular for cases where compressed ARC is enabled. It could also ease future work to allocate pages directly from `segkpm` for minimal-overhead memory allocations, bypassing the `kmem` subsystem.

This is essentially the same change as the one which recently landed in ZFS on Linux, although they made some platform-specific changes while adapting this work to their codebase:

1. Implemented the equivalent of the `segkpm` suggestion for future work mentioned above to bypass issues that they've had with the Linux kernel memory allocator.
2. Changed the internal representation of the ABD's scatter/gather list so it could be used to pass I/O directly into Linux block device drivers. (This feature is not available in the illumos block device interface yet.)


Related issues

Related to illumos gate - Bug #8100: 8021 seems to cause random BAD TRAP: type=d (#gp General protection)Closed2017-04-24

Actions

History

#1

Updated by Daniel Kimmel over 2 years ago

Based on review comments, we decided to switch from 1K to 4K chunks by default. This is tunable at boot via the value of "zfs_abd_chunk_size".

You can also disable chunking up ARC buffers entirely by setting the tunable "zfs_abd_scatter_enabled" to false.

#2

Updated by Daniel Kimmel over 2 years ago

(Also, "zfs_abd_scatter_enabled" can be changed anytime, not just at boot. However, anything present in the cache which is already split into chunks will continue to be stored that way until the buffer has been evicted.)

#3

Updated by Joshua M. Clulow over 2 years ago

  • Subject changed from ABD to ARC buf data scatter-ization
  • Tags deleted (needs-triage)
#4

Updated by Electric Monk over 2 years ago

  • Status changed from New to Closed
  • % Done changed from 0 to 100

git commit 660946868929e02041af7b5b1c3e14f547c53f11

commit  660946868929e02041af7b5b1c3e14f547c53f11
Author: Dan Kimmel <dan.kimmel@delphix.com>
Date:   2017-04-21T17:50:45.000Z

    8021 ARC buf data scatter-ization
    Reviewed by: Matthew Ahrens mahrens@delphix.com
    Reviewed by: George Wilson george.wilson@delphix.com
    Reviewed by: Paul Dagnelie pcd@delphix.com
    Reviewed by: John Kennedy john.kennedy@delphix.com
    Reviewed by: Prakash Surya prakash.surya@delphix.com
    Reviewed by: Prashanth Sreenivasa pks@delphix.com
    Reviewed by: Pavel Zakharov pavel.zakharov@delphix.com
    Reviewed by: Chris Williamson chris.williamson@delphix.com
    Approved by: Richard Lowe <richlowe@richlowe.net>

#5

Updated by Marcel Telka over 2 years ago

  • Related to Bug #8100: 8021 seems to cause random BAD TRAP: type=d (#gp General protection) added
#6

Updated by Electric Monk over 2 years ago

git commit 4ee0199ec059ca9417a72125beae8d89fbc5a9df

commit  4ee0199ec059ca9417a72125beae8d89fbc5a9df
Author: Robert Mustacchi <rm@joyent.com>
Date:   2017-04-25T23:27:23.000Z

    backout: 8021 ARC buf data scatter-ization (causes 8100)

#7

Updated by Electric Monk over 2 years ago

git commit 770499e185d15678ccb0be57ebc626ad18d93383

commit  770499e185d15678ccb0be57ebc626ad18d93383
Author: Dan Kimmel <dan.kimmel@delphix.com>
Date:   2017-05-18T17:15:51.000Z

    8021 ARC buf data scatter-ization
    8100 8021 seems to cause random BAD TRAP: type=d (#gp General protection)
    Reviewed by: Matthew Ahrens mahrens@delphix.com
    Reviewed by: George Wilson george.wilson@delphix.com
    Reviewed by: Paul Dagnelie pcd@delphix.com
    Reviewed by: John Kennedy john.kennedy@delphix.com
    Reviewed by: Prakash Surya prakash.surya@delphix.com
    Reviewed by: Prashanth Sreenivasa pks@delphix.com
    Reviewed by: Pavel Zakharov pavel.zakharov@delphix.com
    Reviewed by: Chris Williamson chris.williamson@delphix.com
    Approved by: Richard Lowe <richlowe@richlowe.net>

Also available in: Atom PDF