Project

General

Profile

Actions

Bug #14375

closed

dmu_zfetch: don't leak unreferenced stream when zfetch is freed

Added by Marcel Telka 7 months ago. Updated 7 months ago.

Status:
Closed
Priority:
Normal
Assignee:
Category:
zfs - Zettabyte File System
Start date:
Due date:
% Done:

100%

Estimated time:
Difficulty:
Medium
Tags:
Gerrit CR:

Description

Let's backport the following fix from openzfs:

commit 57dc5d42b185d2983717580b7fb58bfe423cd834
Author: Matthew Macy <mmacy@freebsd.org>
Date:   Tue Oct 13 21:03:36 2020 -0700

    dmu_zfetch: don't leak unreferenced stream when zfetch is freed

    Currently streams are only freed when:
      - They have no referencing zfetch and and their I/O references
        go to zero.
      - They are more than 2s old and a new I/O request comes in on
        the same zfetch.

    This means that we will leak unreferenced streams when their zfetch
    structure is freed.

    This change checks the reference count on a stream at zfetch free
    time. If it is zero we free it immediately. If it has remaining
    references we allow the prefetch callback to free it at I/O
    completion time.

    Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
    Reviewed-by: Adam Moss <c@yotes.com>
    Reviewed-by: Ryan Moeller <ryan@iXsystems.com>
    Signed-off-by: Matt Macy <mmacy@FreeBSD.org>
    Closes #11052

Related issues

Related to illumos gate - Bug #13374: Port L2ARC Improvements from OpenZFSClosedJason King

Actions
Actions #1

Updated by Marcel Telka 7 months ago

An example of affected machine:

> ::findleaks
findleaks: using cached results (use '-f' to force a full run)
CACHE             LEAKED           BUFCTL CALLER
fffffe2c9ea2c448       1 fffffe2cce70f6a8 AcpiOsAllocate+0x1c
fffffe2c9ea40888       1 fffffe2cf20b59f0 allocb+0x50
fffffe2cceaa5888       1 fffffe2dc4860238 cralloc_flags+0x17
fffffe2c9ea2a008       1 fffffe2d12129540 crgrpcopyin+0x41
fffffe2c9ea40008       1 fffffe2cf2040a88 dblk_constructor+0x46
fffffe2c9ea2e008      15 fffffe2ce93cd698 ddi_umem_alloc+0x68
fffffe2c9ea2e008       2 fffffe2d2d949490 ddi_umem_alloc+0x68
fffffe2c9ea2e008       1 fffffe2d2d33b340 ddi_umem_alloc+0x68
fffffe2c9ea2e008       1 fffffe2d39c8e9c8 ddi_umem_alloc+0x68
fffffe2c9ea2e008       1 fffffe2d39395698 ddi_umem_alloc+0x68
fffffe2c9ea2e008       1 fffffe2cdcf674d8 ddi_umem_alloc+0x68
fffffe2c9ea2e008       1 fffffe2d3961b750 ddi_umem_alloc+0x68
fffffe2c9ea2e008       1 fffffe2cf1935bb0 ddi_umem_alloc+0x68
fffffe2c9ea2c448       3 fffffe2e87052cf8 dmu_zfetch_stream_create+0x104
fffffe2c9ea2c448    1123 fffffe3661e740a8 dmu_zfetch_stream_create+0x104
fffffe2c9ea2c448   43652 fffffe2d14c21cd8 dmu_zfetch_stream_create+0x104
fffffe2c9ea2c448       3 fffffe2d30547a40 dmu_zfetch_stream_create+0x104
fffffe2c9ea2c448       2 fffffe2cdfa7e268 dmu_zfetch_stream_create+0x104
fffffe2c9ea2c448    3645 fffffe2d4f1a31b0 dmu_zfetch_stream_create+0x104
fffffe2c9ea2c448  311940 fffffe36ad520370 dmu_zfetch_stream_create+0x104
fffffe2c9ea2c448       5 fffffe2d40f8f4d8 dmu_zfetch_stream_create+0x104
fffffe2c9ea2c448       3 fffffe2d148f4380 dmu_zfetch_stream_create+0x104
fffffe2c9ea2c448       8 fffffe2ce28811a0 dmu_zfetch_stream_create+0x104
fffffe2c9ea2c448       6 fffffe2d4df62350 dmu_zfetch_stream_create+0x104
fffffe2c9ea2c448     603 fffffe2d162555d0 dmu_zfetch_stream_create+0x104
fffffe2c9ea2c448       3 fffffe2cdb698490 dmu_zfetch_stream_create+0x104
fffffe2c9ea2c448       1 fffffe466dabfe38 dmu_zfetch_stream_create+0x104
fffffe2c9ea2c448     184 fffffe2d162d5ac8 dmu_zfetch_stream_create+0x104
fffffe2c9ea2c448       2 fffffe3f5d4c6ee0 dmu_zfetch_stream_create+0x104
fffffe2c9ea2c448       1 fffffe2e809c0dd8 dmu_zfetch_stream_create+0x104
fffffe2c9ea2c448       2 fffffe2d14613748 dmu_zfetch_stream_create+0x104
fffffe2c9ea2c448      11 fffffe2d52039e58 dmu_zfetch_stream_create+0x104
fffffe2c9ea2c448       5 fffffe2d3b8bfcf0 dmu_zfetch_stream_create+0x104
fffffe2c9ea2c448       5 fffffe4ef0f597f0 dmu_zfetch_stream_create+0x104
fffffe2c9ea2c448       9 fffffe2cdd10ed80 dmu_zfetch_stream_create+0x104
fffffe2c9ea2c448       1 fffffe2cded777f0 dmu_zfetch_stream_create+0x104
fffffe2c9ea2c448    3035 fffffe36621f7e40 dmu_zfetch_stream_create+0x104
fffffe2c9ea2c448       1 fffffe2d147b1970 dmu_zfetch_stream_create+0x104
fffffe2c9ea2c448  199042 fffffe2e3be54750 dmu_zfetch_stream_create+0x104
fffffe2c9ea2c448       2 fffffe2cda6e7db8 dmu_zfetch_stream_create+0x104
fffffe2c9ea2c448     121 fffffe2d15d06038 dmu_zfetch_stream_create+0x104
fffffe2c9ea2c448    1482 fffffe3ce50a1090 dmu_zfetch_stream_create+0x104
fffffe2c9ea2c448       2 fffffe2d14c229e0 dmu_zfetch_stream_create+0x104
fffffe2c9ea2c448       7 fffffe2d410980e8 dmu_zfetch_stream_create+0x104
fffffe2c9ea2c448      40 fffffe2d146005f0 dmu_zfetch_stream_create+0x104
fffffe2c9ea2c448       2 fffffe2d0dab7518 dmu_zfetch_stream_create+0x104
fffffe2c9ea2c448       8 fffffe2d16118ac8 dmu_zfetch_stream_create+0x104
fffffe2c9ea2c448       4 fffffe2d51281800 dmu_zfetch_stream_create+0x104
fffffe2c9ea2c448     724 fffffe2d1488c200 dmu_zfetch_stream_create+0x104
fffffe2c9ea2c448       2 fffffe4b5be116f0 dmu_zfetch_stream_create+0x104
fffffe2c9ea2c448    1156 fffffe2d51a7e9c8 dmu_zfetch_stream_create+0x104
fffffe2c9ea2c448   80122 fffffe2d15dad268 dmu_zfetch_stream_create+0x104
fffffe2c9ea2c448       1 fffffe2cdd10d178 dmu_zfetch_stream_create+0x104
fffffe2c9ea2c448       6 fffffe2cd6eb05a0 dmu_zfetch_stream_create+0x104
fffffe2c9ea2c448      21 fffffe2d14ffddb0 dmu_zfetch_stream_create+0x104
fffffe2c9ea2c448       1 fffffe2cdc99ebd0 dmu_zfetch_stream_create+0x104
fffffe2c9ea2c448       2 fffffe2d15cec3f0 dmu_zfetch_stream_create+0x104
fffffe2c9ea2e008       2 fffffe2d3acb0420 rtR0MemAllocEx+0x64
fffffe2c9ea2e008       1 fffffe2d3bcb1858 rtR0MemAllocEx+0x64
fffffe2c9ea2e008       1 fffffe2d3e1cac78 rtR0MemAllocEx+0x64
fffffe2c9ea2e008      15 fffffe2ce19f08e8 rtR0MemAllocEx+0x64
fffffe2c9ea2e008      15 fffffe2cf1934128 rtR0MemAllocEx+0x64
fffffe2c9ea2e008       1 fffffe2d3b141da8 rtR0MemAllocEx+0x64
fffffe2c9ea2e008       1 fffffe2cf42af710 rtR0MemAllocEx+0x64
fffffe2c9ea2e008       1 fffffe2cf19331e0 rtR0MemAllocEx+0x64
fffffe2c9ea2e008       1 fffffe2cf19347e8 rtR0MemAllocEx+0x64
fffffe2c9ea2e008       1 fffffe2d15555958 rtR0MemAllocEx+0x64
fffffe2c9ea2e008       1 fffffe2d3bb687d8 rtR0MemAllocEx+0x64
fffffe2c9ea2e008       2 fffffe2d385b54a8 rtR0MemAllocEx+0x64
fffffe2c9ea2e008       1 fffffe2ce8de7e20 rtR0MemAllocEx+0x64
fffffe2c9ea2e008       1 fffffe2cf21154e8 rtR0MemAllocEx+0x64
fffffe2c9ea2e008       1 fffffe2cde998b10 rtR0MemAllocEx+0x64
fffffe2c9ea2c008       1 fffffe2d6ec1ab30 rtR0MemAllocEx+0x78
fffffe2c9ea30008       1 fffffe2d59ba9360 rtR0MemAllocEx+0x78
fffffe2c9ea30008      15 fffffe2d59ba44b8 rtR0MemAllocEx+0x78
fffffe2c9ea30008       1 fffffe2d59ba6948 rtR0MemAllocEx+0x78
fffffe2c9ea30008       1 fffffe2ce9a5c638 rtR0MemAllocEx+0x78
fffffe2c9ea2b008       1 fffffe2d790757e0 rtR0MemAllocEx+0x78
fffffe2c9ea30008       1 fffffe2d44528e48 rtR0MemAllocEx+0x78
fffffe2c9ea30008       1 fffffe2d190cdb60 rtR0MemAllocEx+0x78
------------------------------------------------------------------------
           Total  647093 buffers, 51774496 bytes
> panic_lbolt64/D
panic_lbolt64:
panic_lbolt64:  3938210
> usec_per_tick/D
usec_per_tick:
usec_per_tick:  1000
>

IOW, it leaks ca 50 MB/hour.

Actions #2

Updated by Marcel Telka 7 months ago

Testing

Without the fix

# while true ; do echo -n "$(date +%'[%Y-%m-%d %H:%M:%S] ')" ; mdb -ke '::kmem_slabs -N kmem_alloc_80' | grep "^kmem" ; sleep 600 ; done
[2022-01-08 07:20:34] kmem_alloc_80                60595        1   3029750        10   0.0%
[2022-01-08 07:30:35] kmem_alloc_80                63007        1   3150350        31   0.0%
[2022-01-08 07:40:35] kmem_alloc_80                63150        1   3157500        15   0.0%
[2022-01-08 07:50:36] kmem_alloc_80                66004        1   3300200        23   0.0%
[2022-01-08 08:00:37] kmem_alloc_80                66121        2   3306050        51   0.0%
[2022-01-08 08:10:37] kmem_alloc_80                70191        1   3509550        46   0.0%
[2022-01-08 08:20:38] kmem_alloc_80                72681        1   3634050        45   0.0%
[2022-01-08 08:30:39] kmem_alloc_80                74632        1   3731600        31   0.0%
[2022-01-08 08:40:39] kmem_alloc_80                75327        1   3766350        39   0.0%
[2022-01-08 08:50:40] kmem_alloc_80                78174        1   3908700         3   0.0%
[2022-01-08 09:00:40] kmem_alloc_80                79270        1   3963500        10   0.0%
[2022-01-08 09:10:41] kmem_alloc_80                81167        1   4058350        15   0.0%
[2022-01-08 09:20:42] kmem_alloc_80                84112        1   4205600        38   0.0%

With the fix

# while true ; do echo -n "$(date +%'[%Y-%m-%d %H:%M:%S] ')" ; mdb -ke '::kmem_slabs -N kmem_alloc_80' | grep "^kmem" ; sleep 600 ; done
[2022-01-08 14:17:51] kmem_alloc_80                 1011        1     50550        37   0.1%
[2022-01-08 14:27:52] kmem_alloc_80                 1011        1     50550        37   0.1%
[2022-01-08 14:37:53] kmem_alloc_80                 1011        1     50550        37   0.1%
[2022-01-08 14:47:53] kmem_alloc_80                 1011        1     50550        37   0.1%
[2022-01-08 14:57:54] kmem_alloc_80                 1011        1     50550        37   0.1%
[2022-01-08 15:07:55] kmem_alloc_80                 1011        1     50550        37   0.1%
[2022-01-08 15:17:55] kmem_alloc_80                 1043        1     52150        34   0.1%
[2022-01-08 15:27:56] kmem_alloc_80                 1043        1     52150        34   0.1%
[2022-01-08 15:37:57] kmem_alloc_80                 1043        1     52150        34   0.1%
[2022-01-08 15:47:58] kmem_alloc_80                 1043        1     52150        34   0.1%
[2022-01-08 15:57:58] kmem_alloc_80                 1043        1     52150        34   0.1%
[2022-01-08 16:07:59] kmem_alloc_80                 1043        1     52150        34   0.1%
[2022-01-08 16:18:00] kmem_alloc_80                 1043        1     52150        34   0.1%
Actions #3

Updated by Marcel Telka 7 months ago

  • Related to Bug #13374: Port L2ARC Improvements from OpenZFS added
Actions #4

Updated by Electric Monk 7 months ago

  • Gerrit CR set to 1926
Actions #5

Updated by Marcel Telka 7 months ago

  • Status changed from In Progress to Pending RTI
Actions #6

Updated by Electric Monk 7 months ago

  • Status changed from Pending RTI to Closed
  • % Done changed from 0 to 100

git commit 2966a4e8bbd82f7bd6a8909e7c10b98f18fe7e84

commit  2966a4e8bbd82f7bd6a8909e7c10b98f18fe7e84
Author: Matt Macy <mmacy@FreeBSD.org>
Date:   2022-01-11T16:18:51.000Z

    14375 dmu_zfetch: don't leak unreferenced stream when zfetch is freed
    Reviewed by: Brian Behlendorf <behlendorf1@llnl.gov>
    Reviewed by: Adam Moss <c@yotes.com>
    Reviewed by: Ryan Moeller <ryan@iXsystems.com>
    Reviewed by: Marcel Telka <marcel@telka.sk>
    Reviewed by: Toomas Soome <tsoome@me.com>
    Reviewed by: Andy Fiddaman <andy@omnios.org>
    Approved by: Dan McDonald <danmcd@joyent.com>

Actions

Also available in: Atom PDF