Project

General

Profile

Bug #9339

ztest process hung remapping blocks from a removed vdev

Added by Brad Lewis over 1 year ago.

Status:
New
Priority:
Normal
Assignee:
-
Category:
-
Start date:
2018-03-23
Due date:
% Done:

0%

Estimated time:
Difficulty:
Medium
Tags:
needs-triage

Description

We encountered a hung zloop run, where ztest was consuming a cpu, but not making progress. spa_sync was stuck like this:

stack pointer for thread 15b: fffffd7fd9a95ad0
[ fffffd7fd9a95ad0 libc.so.1`__lwp_park+0x17() ]
fffffd7fd9a95b30 libc.so.1`cond_wait_queue+0x5b(1d3b288, 1d3b220, 0)
fffffd7fd9a95b80 libc.so.1`__cond_wait+0xb3(1d3b288, 1d3b220)
fffffd7fd9a95bb0 libc.so.1`cond_wait+0x2a(1d3b288, 1d3b220)
fffffd7fd9a95bf0 libcmdutils.so.1`utaskq_wait+0x53(1d3b200)
fffffd7fd9a95cf0 libzpool.so.1`dmu_objset_sync+0x28b(29277c0, 19cca000, 19dfdd00)
fffffd7fd9a95d70 libzpool.so.1`dsl_dataset_sync+0xb8(21ce400, 19cca000, 19dfdd00)
fffffd7fd9a95e00 libzpool.so.1`dsl_pool_sync+0xac(1e82740, 1ea)
fffffd7fd9a95ef0 libzpool.so.1`spa_sync+0x449(1d2f000, 1ea)
fffffd7fd9a95fb0 libzpool.so.1`txg_sync_thread+0x273(1e82740)
fffffd7fd9a95fe0 libc.so.1`_thrp_setup+0x8a(fffffd7ffead3a40)
fffffd7fd9a95ff0 libc.so.1`_lwp_start()

stack pointer for thread 12e: fffffd7fd9498d50
[ fffffd7fd9498d50 libc.so.1`__lwp_park+0x17() ]
fffffd7fd9498d90 libc.so.1`rw_rdlock_impl+0x204(14fd51e0, 0)
fffffd7fd9498dc0 libc.so.1`pthread_rwlock_rdlock+0x45(14fd51e0)
fffffd7fd9498e00 libzpool.so.1`rw_enter+0x8a(14fd51d0, 0)
fffffd7fd9498e40 libzpool.so.1`dnode_verify+0x3c5(14fd51d0)
fffffd7fd9498ed0 libzpool.so.1`dnode_sync+0x94(14fd51d0, 19dfdd00)
fffffd7fd9498f10 libzpool.so.1`dmu_objset_sync_dnodes+0x99(2157920, 19dfdd00)
fffffd7fd9498f40 libzpool.so.1`sync_dnodes_task+0x31(1460b6c0)
fffffd7fd9498fb0 libcmdutils.so.1`utaskq_thread+0xe0(1d3b200)
fffffd7fd9498fe0 libc.so.1`_thrp_setup+0x8a(fffffd7ffead5240)
fffffd7fd9498ff0 libc.so.1`_lwp_start()

and the owner of the rwlock:

stack pointer for thread 177: fffffd7fd4cbccc0
[ fffffd7fd4cbccc0 libc.so.1`__pollsys+0xa() ]
fffffd7fd4cbcd00 libc.so.1`poll+0x56(0, 0, 0)
fffffd7fd4cbcd20 libzpool.so.1`delay+0x1c(0)
fffffd7fd4cbcd80 libzpool.so.1`dmu_object_remap_one_indirect+0xce(29277c0, 14fd51d0, 1a1, 4b5f398e000000)
fffffd7fd4cbcdf0 libzpool.so.1`dmu_object_remap_indirects+0x9b(29277c0, 29, 1a1)
fffffd7fd4cbce40 libzpool.so.1`dmu_objset_remap_indirects_impl+0x52(29277c0, 1a1)
fffffd7fd4cbce90 libzpool.so.1`dmu_objset_remap_indirects+0x18f(48db50)
fffffd7fd4cbcec0 ztest_remap_blocks+0x2a(48daf0, 10)
fffffd7fd4cbcf50 ztest_execute+0x83(1e, 420f50, 10)
fffffd7fd4cbcfb0 ztest_thread+0xf4(10)
fffffd7fd4cbcfe0 libc.so.1`_thrp_setup+0x8a(fffffd7ffeae4240)
fffffd7fd4cbcff0 libc.so.1`_lwp_start()

It seems possible that we're inifinite looping in the while loop at the bottom of dmu_object_remap_indirects().
If dnode_next_offset() is called with minlvl > dn_phys->dn_nlevels, it returns 0 without advancing *offset. This should be fixed, probably to return ESRCH.

The interesting thing is why we called dnode_next_offset() with minlvl > dn_phys->dn_nlevels. dmu_object_remap_indirects() doesn't call dnode_next_offset(minlvl=2) "If the dnode has no indirect blocks". However, it's checking that based on dn->dn_nlevels (not dn_phys->dn_nlevels). So if we have increased dn_nlevels but not yet synced it out to dn_phys->dn_nlevels, we can get into this situation.

Also available in: Atom PDF