Bug #8521

nvlist memory leak in get_clones_stat() and spa_load_best()

Added by Prakash Surya 2 months ago. Updated about 2 months ago.

Status:ClosedStart date:2017-07-21
Priority:NormalDue date:
Assignee:Prakash Surya% Done:

100%

Category:zfs - Zettabyte File System
Target version:-
Difficulty:Medium Tags:needs-triage

Description

Yuri reported this to the mailing list:

doing a `reboot -d` on current illumos-gate HEAD gives the following "::findleaks -dv" output:

findleaks: maximum buffers => 301061
findleaks: actual buffers => 297587
findleaks:
findleaks: potential pointers => 29289774
findleaks: dismissals => 26242305 (89.5%)
findleaks: misses => 331153 ( 1.1%)
findleaks: dups => 2419681 ( 8.2%)
findleaks: follows => 296635 ( 1.0%)
findleaks:
findleaks: peak memory usage => 7353 kB
findleaks: elapsed CPU time => 1.5 seconds
findleaks: elapsed wall time => 2.0 seconds
findleaks:
CACHE LEAKED BUFCTL CALLER
ffffff03d222b008 120 ffffff03ef7ceb78 nv_alloc_sys+0x1f
ffffff03d222a448 123 ffffff03f4150cc8 nv_alloc_sys+0x1f
ffffff03d222b448 5 ffffff03f28bd598 nv_alloc_sys+0x1f
ffffff03d222b888 87 ffffff03f28c10f0 nv_alloc_sys+0x1f
ffffff03d222c008 21 ffffff03f4139310 nv_alloc_sys+0x1f
ffffff03d222b888 43 ffffff040ef3f3e8 nv_alloc_sys+0x1f
ffffff03d222c008 120 ffffff03f4591e58 nv_alloc_sys+0x1f
ffffff03d222b008 121 ffffff03f352c068 nv_alloc_sys+0x1f
ffffff03d222a448 112 ffffff03f414e5f8 nv_alloc_sys+0x1f
ffffff03d222b008 119 ffffff03ee92fdc0 nv_alloc_sys+0x1f
ffffff03d222b888 46 ffffff03f28c1378 nv_alloc_sys+0x1f
ffffff03d222b448 4 ffffff03f28c7708 nv_alloc_sys+0x1f
ffffff03d222c008 20 ffffff03f2a6e7e8 nv_alloc_sys+0x1f
ffffff03d222a448 11 ffffff03f414f018 nv_alloc_sys+0x1f
------------------------------------------------------------------------
Total 952 buffers, 40896 bytes

kmem_alloc_40 leak: 120 buffers, 40 bytes each, 4800 bytes total
ADDR BUFADDR TIMESTAMP THREAD
CACHE LASTLOG CONTENTS
ffffff03ef7ceb78 ffffff03ed8e8c80 ab5dc8356 ffffff03f3b97100
ffffff03d222b008 ffffff03d9d4f5c0 ffffff03e57574d8
kmem_cache_alloc_debug+0x2e0
kmem_cache_alloc+0xdd
kmem_alloc+0x4b
nv_alloc_sys+0x1f
nv_mem_zalloc+0x21
nv_priv_alloc_embedded+0x1d
nvlist_copy_embedded+0x2b
nvlist_add_common+0x36d
nvlist_add_nvlist+0x26
fnvlist_add_nvlist+0x19
get_clones_stat+0x83
dsl_dataset_stats+0xcc
dmu_objset_stats+0x2e
zfs_ioc_objset_stats_impl+0x64
zfs_ioc_snapshot_list_next+0x191
kmem_alloc_24 leak: 123 buffers, 24 bytes each, 2952 bytes total
ADDR BUFADDR TIMESTAMP THREAD
CACHE LASTLOG CONTENTS
ffffff03f4150cc8 ffffff03f43b29c0 ab5d77f2b ffffff03f3b97100
ffffff03d222a448 ffffff03d9d4a100 ffffff03e5756558
kmem_cache_alloc_debug+0x2e0
kmem_cache_alloc+0xdd
kmem_alloc+0x4b
nv_alloc_sys+0x1f
nv_mem_zalloc+0x21
nvlist_xalloc+0x67
nvlist_alloc+0x2a
fnvlist_alloc+0x18
get_clones_stat+0x27
dsl_dataset_stats+0xcc
dmu_objset_stats+0x2e
zfs_ioc_objset_stats_impl+0x64
zfs_ioc_snapshot_list_next+0x191
zfsdev_ioctl+0x546
cdev_ioctl+0x39

kmem_alloc_48 leak: 5 buffers, 48 bytes each, 240 bytes total
ADDR BUFADDR TIMESTAMP THREAD
CACHE LASTLOG CONTENTS
ffffff03f28bd598 ffffff03f28e5390 acfd4f020 ffffff03f3b97100
ffffff03d222b448 ffffff03db98a980 ffffff03de82d840
kmem_cache_alloc_debug+0x2e0
kmem_cache_alloc+0xdd
kmem_alloc+0x4b
nv_alloc_sys+0x1f
nv_mem_zalloc+0x21
nvp_buf_alloc+0x24
nvlist_add_common+0x11c
nvlist_add_boolean+0x1f
fnvlist_add_boolean+0x15
get_clones_stat_impl+0x151
get_clones_stat+0x4b
dsl_dataset_stats+0xcc
dmu_objset_stats+0x2e
zfs_ioc_objset_stats_impl+0x64
zfs_ioc_snapshot_list_next+0x191
kmem_alloc_56 leak: 87 buffers, 56 bytes each, 4872 bytes total
ADDR BUFADDR TIMESTAMP THREAD
CACHE LASTLOG CONTENTS
ffffff03f28c10f0 ffffff03f28e4a58 aba805885 ffffff03f3b97100
ffffff03d222b888 ffffff03d8ae5900 ffffff03e450d260
kmem_cache_alloc_debug+0x2e0
kmem_cache_alloc+0xdd
kmem_alloc+0x4b
nv_alloc_sys+0x1f
nv_mem_zalloc+0x21
nvp_buf_alloc+0x24
nvlist_add_common+0x11c
nvlist_add_boolean+0x1f
fnvlist_add_boolean+0x15
get_clones_stat_impl+0x151
get_clones_stat+0x4b
dsl_dataset_stats+0xcc
dmu_objset_stats+0x2e
zfs_ioc_objset_stats_impl+0x64
zfs_ioc_snapshot_list_next+0x191

kmem_alloc_64 leak: 21 buffers, 64 bytes each, 1344 bytes total
ADDR BUFADDR TIMESTAMP THREAD
CACHE LASTLOG CONTENTS
ffffff03f4139310 ffffff03f43c4500 ab5dbac69 ffffff03f3b97100
ffffff03d222c008 ffffff03d9d4ed80 ffffff03e5749298
kmem_cache_alloc_debug+0x2e0
kmem_cache_alloc+0xdd
kmem_alloc+0x4b
nv_alloc_sys+0x1f
nv_mem_zalloc+0x21
nvp_buf_alloc+0x24
nvlist_add_common+0x11c
nvlist_add_boolean+0x1f
fnvlist_add_boolean+0x15
get_clones_stat_impl+0x151
get_clones_stat+0x4b
dsl_dataset_stats+0xcc
dmu_objset_stats+0x2e
zfs_ioc_objset_stats_impl+0x64
zfs_ioc_snapshot_list_next+0x191
kmem_alloc_56 leak: 43 buffers, 56 bytes each, 2408 bytes total
ADDR BUFADDR TIMESTAMP THREAD
CACHE LASTLOG CONTENTS
ffffff040ef3f3e8 ffffff040fbc81f0 c8f9dddd9 ffffff040b1a23e0
ffffff03d222b888 ffffff03d7e13480 0
kmem_cache_alloc_debug+0x2e0
kmem_cache_alloc+0x320
kmem_alloc+0x4b
nv_alloc_sys+0x1f
nv_mem_zalloc+0x21
nvp_buf_alloc+0x24
nvlist_add_common+0x11c
nvlist_copy_pairs+0x64
nvlist_copy_embedded+0x50
nvlist_add_common+0x36d
nvlist_add_nvlist+0x26
fnvlist_add_nvlist+0x19
get_clones_stat+0x83
dsl_dataset_stats+0xcc
dmu_objset_stats+0x2e

kmem_alloc_64 leak: 120 buffers, 64 bytes each, 7680 bytes total
ADDR BUFADDR TIMESTAMP THREAD
CACHE LASTLOG CONTENTS
ffffff03f4591e58 ffffff03f457be00 ab5dc7f46 ffffff03f3b97100
ffffff03d222c008 ffffff03d9d4f500 ffffff03e57491d0
kmem_cache_alloc_debug+0x2e0
kmem_cache_alloc+0xdd
kmem_alloc+0x4b
nv_alloc_sys+0x1f
nv_mem_zalloc+0x21
nvp_buf_alloc+0x24
nvlist_add_common+0x11c
nvlist_add_nvlist+0x26
fnvlist_add_nvlist+0x19
get_clones_stat+0x83
dsl_dataset_stats+0xcc
dmu_objset_stats+0x2e
zfs_ioc_objset_stats_impl+0x64
zfs_ioc_snapshot_list_next+0x191
zfsdev_ioctl+0x546
kmem_alloc_40 leak: 121 buffers, 40 bytes each, 4840 bytes total
ADDR BUFADDR TIMESTAMP THREAD
CACHE LASTLOG CONTENTS
ffffff03f352c068 ffffff03f34bc840 ab5d77d47 ffffff03f3b97100
ffffff03d222b008 ffffff03d9d4a040 ffffff03e5756570
kmem_cache_alloc_debug+0x2e0
kmem_cache_alloc+0xdd
kmem_alloc+0x4b
nv_alloc_sys+0x1f
nv_priv_alloc+0x1e
nvlist_xalloc+0x4d
nvlist_alloc+0x2a
fnvlist_alloc+0x18
get_clones_stat+0x27
dsl_dataset_stats+0xcc
dmu_objset_stats+0x2e
zfs_ioc_objset_stats_impl+0x64
zfs_ioc_snapshot_list_next+0x191
zfsdev_ioctl+0x546
cdev_ioctl+0x39

kmem_alloc_24 leak: 112 buffers, 24 bytes each, 2688 bytes total
ADDR BUFADDR TIMESTAMP THREAD
CACHE LASTLOG CONTENTS
ffffff03f414e5f8 ffffff03f43b2480 abec5ac72 ffffff03f3b97100
ffffff03d222a448 ffffff03d5331700 ffffff03e302c208
kmem_cache_alloc_debug+0x2e0
kmem_cache_alloc+0xdd
kmem_alloc+0x4b
nv_alloc_sys+0x1f
nv_mem_zalloc+0x21
nvlist_xalloc+0x67
nvlist_alloc+0x2a
get_clones_stat+0x37
dsl_dataset_stats+0xcc
dmu_objset_stats+0x2e
zfs_ioc_objset_stats_impl+0x64
zfs_ioc_snapshot_list_next+0x191
zfsdev_ioctl+0x546
cdev_ioctl+0x39
spec_ioctl+0x60
kmem_alloc_40 leak: 119 buffers, 40 bytes each, 4760 bytes total
ADDR BUFADDR TIMESTAMP THREAD
CACHE LASTLOG CONTENTS
ffffff03ee92fdc0 ffffff03eeee2dc0 ab5d782a6 ffffff03f3b97100
ffffff03d222b008 ffffff03d9d4a1c0 ffffff03e57490d0
kmem_cache_alloc_debug+0x2e0
kmem_cache_alloc+0xdd
kmem_alloc+0x4b
nv_alloc_sys+0x1f
nv_priv_alloc+0x1e
nvlist_xalloc+0x4d
nvlist_alloc+0x2a
get_clones_stat+0x37
dsl_dataset_stats+0xcc
dmu_objset_stats+0x2e
zfs_ioc_objset_stats_impl+0x64
zfs_ioc_snapshot_list_next+0x191
zfsdev_ioctl+0x546
cdev_ioctl+0x39
spec_ioctl+0x60

kmem_alloc_56 leak: 46 buffers, 56 bytes each, 2576 bytes total
ADDR BUFADDR TIMESTAMP THREAD
CACHE LASTLOG CONTENTS
ffffff03f28c1378 ffffff03f28e4968 aba80e3f6 ffffff03f3b97100
ffffff03d222b888 ffffff03d8ae6200 ffffff03e2945e98
kmem_cache_alloc_debug+0x2e0
kmem_cache_alloc+0xdd
kmem_alloc+0x4b
nv_alloc_sys+0x1f
nv_mem_zalloc+0x21
nvp_buf_alloc+0x24
nvlist_add_common+0x11c
nvlist_copy_pairs+0x64
nvlist_copy_embedded+0x50
nvlist_add_common+0x36d
nvlist_add_nvlist+0x26
fnvlist_add_nvlist+0x19
get_clones_stat+0x83
dsl_dataset_stats+0xcc
dmu_objset_stats+0x2e
kmem_alloc_48 leak: 4 buffers, 48 bytes each, 192 bytes total
ADDR BUFADDR TIMESTAMP THREAD
CACHE LASTLOG CONTENTS
ffffff03f28c7708 ffffff03f28e27d8 acfd5c6b6 ffffff03f3b97100
ffffff03d222b448 ffffff03db98b280 ffffff03de838288
kmem_cache_alloc_debug+0x2e0
kmem_cache_alloc+0xdd
kmem_alloc+0x4b
nv_alloc_sys+0x1f
nv_mem_zalloc+0x21
nvp_buf_alloc+0x24
nvlist_add_common+0x11c
nvlist_copy_pairs+0x64
nvlist_copy_embedded+0x50
nvlist_add_common+0x36d
nvlist_add_nvlist+0x26
fnvlist_add_nvlist+0x19
get_clones_stat+0x83
dsl_dataset_stats+0xcc
dmu_objset_stats+0x2e

kmem_alloc_64 leak: 20 buffers, 64 bytes each, 1280 bytes total
ADDR BUFADDR TIMESTAMP THREAD
CACHE LASTLOG CONTENTS
ffffff03f2a6e7e8 ffffff03f3ee4280 ab5dc893d ffffff03f3b97100
ffffff03d222c008 ffffff03d9d4f680 ffffff03e5748cd0
kmem_cache_alloc_debug+0x2e0
kmem_cache_alloc+0xdd
kmem_alloc+0x4b
nv_alloc_sys+0x1f
nv_mem_zalloc+0x21
nvp_buf_alloc+0x24
nvlist_add_common+0x11c
nvlist_copy_pairs+0x64
nvlist_copy_embedded+0x50
nvlist_add_common+0x36d
nvlist_add_nvlist+0x26
fnvlist_add_nvlist+0x19
get_clones_stat+0x83
dsl_dataset_stats+0xcc
dmu_objset_stats+0x2e

kmem_alloc_24 leak: 11 buffers, 24 bytes each, 264 bytes total
ADDR BUFADDR TIMESTAMP THREAD
CACHE LASTLOG CONTENTS
ffffff03f414f018 ffffff03f43b2930 ab5d787f9 ffffff03f3b97100
ffffff03d222a448 ffffff03d9d4a280 0
kmem_cache_alloc_debug+0x2e0
kmem_cache_alloc+0x320
kmem_alloc+0x4b
nv_alloc_sys+0x1f
nv_mem_zalloc+0x21
nvlist_xalloc+0x67
nvlist_alloc+0x2a
get_clones_stat+0x37
dsl_dataset_stats+0xcc
dmu_objset_stats+0x2e
zfs_ioc_objset_stats_impl+0x64
zfs_ioc_snapshot_list_next+0x191
zfsdev_ioctl+0x546
cdev_ioctl+0x39
spec_ioctl+0x60

History

#1 Updated by Prakash Surya about 2 months ago

A couple things to make note of:

1. nvlist_add_nvlist() (and the "fnvlist" variant) copies the passed-in nvlist leaving it to the caller to free any passed-in nvlist.

2. spa_config_set() does NOT copy in its nvlist (but we missed a free in spa_load_best() anyway).

#2 Updated by Electric Monk about 2 months ago

  • % Done changed from 0 to 100
  • Status changed from New to Closed

git commit 7d3000f774e20097a1ee45cbd06d0e38065ddd5a

commit  7d3000f774e20097a1ee45cbd06d0e38065ddd5a
Author: Pavel Zakharov <pavel.zakharov@delphix.com>
Date:   2017-08-01T14:47:35.000Z

    8521 nvlist memory leak in get_clones_stat() and spa_load_best()
    Reviewed by: Steve Gonczi <steve.gonczi@delphix.com>
    Reviewed by: George Wilson <george.wilson@delphix.com>
    Reviewed by: Yuri Pankov <yuripv@gmx.com>
    Reviewed by: Matt Ahrens <mahrens@delphix.com>
    Approved by: Dan McDonald <danmcd@joyent.com>

Also available in: Atom