Project

General

Profile

Actions
Copy link

Bug #5543

open

zfs lz4 compression cause two cross-calls per I/O

Added by Alexander Kolbasov over 8 years ago. Updated over 8 years ago.

Status:
New
Priority:
Normal
Assignee:
-
Category:
-
Start date:
2015-01-17
Due date:
% Done:

0%

Estimated time:
Difficulty:
Medium
Tags:
needs-triage
Gerrit CR:
External Bug:

Description

When lz4 compression is enabled and uses the "heap" mode it goes through the following code:

static int
real_LZ4_compress(const char *source, char *dest, int isize, int osize)
{
    void *ctx = kmem_zalloc(sizeof (struct refTables), KM_NOSLEEP);
    int result;

    /*
     * out of kernel memory, gently fall through - this will disable
     * compression in zio_compress_data
     */
    if (ctx == NULL)
        return (0);

    if (isize < LZ4_64KLIMIT)
        result = LZ4_compress64kCtx(ctx, source, dest, isize, osize);
    else
        result = LZ4_compressCtx(ctx, source, dest, isize, osize);

    kmem_free(ctx, sizeof (struct refTables));
    return (result);
}

When the size is 16 K every allocation goes creates a new club and every kmem_free destroys it. Each club allocation/destruction causes TLB shutdown which is very expensive and we have two club operations per I/O which isn't good.
Actions
Copy link
 #1

Updated by Josef Sipek over 8 years ago

Recently, there was a change that should turn the kmem_free into at most 1 cross-call. Does that help? If not, can you get us the stacks for the cross calls you're talking about? (DTrace is your friend.)

commit a6a74e0e62d62ff750cd4b790be5eacc99c3bb8c
Author: Matthew Ahrens <mahrens@delphix.com>
Date:   Thu Jan 15 18:21:03 2015 -0800

    5498 kmem_reap does one xcall per page
    5514 hat_unload_callback passes the wrong length to segvn_hat_unload_callback
    Reviewed by: Adam Leventhal <ahl@delphix.com>
    Reviewed by: Dan Kimmel <dan.kimmel@delphix.com>
    Reviewed by: George Wilson <george.wilson@delphix.com>
    Reviewed by: Josef 'Jeff' Sipek <josef.sipek@nexenta.com>
    Reviewed by: Paul Dagnelie <paul.dagnelie@delphix.com>
    Reviewed by: Robert Mustacchi <rm@joyent.com>
    Approved by: Richard Lowe <richlowe@richlowe.net>
<pre>
Actions
Copy link

Also available in: Atom PDF