Bug #8493
kmem_move taskq appears to be inducing significant system latency
Start date:
2017-07-14
Due date:
% Done:
100%
Estimated time:
Difficulty:
Hard
Tags:
Gerrit CR:
Description
Note this is a change that logically combines Joyent's OS-4162 and OS-4949. The analysis from both tickets is included here.
One of the Manta storage systems in us-east got into a state today where the console hung after typing the username, ssh to the box was hanging, and sdc-oneachnode timed out trying to find the node. However, the system wasn't totally wedged: the marlin agent was making good forward progress (accepting tasks in as little as 80ms). I also had a shell on the system from which I was able to run several commands, including mpstat, which showed the box was largely idle:
[root@3d1e2fc2 (storage) ~]$ mpstat 1 CPU minf mjf xcal intr ithr csw icsw migr smtx srw syscl usr sys wt idl 0 7899 0 1628 3466 138 1048 17 132 619 0 5466 8 22 0 70 1 3786 0 1441 1246 44 941 13 96 610 0 3867 5 14 0 81 2 2318 0 910 908 25 653 9 70 525 0 2547 4 10 0 86 3 1106 0 739 886 42 800 9 55 512 0 2576 4 7 0 88 4 995 0 557 763 101 543 8 41 448 0 1874 3 6 0 91 5 836 0 610 841 101 760 8 41 488 0 2409 4 5 0 90 6 942 0 505 679 88 485 7 33 429 0 1722 3 5 0 92 7 731 0 557 815 106 721 8 36 466 0 2256 4 5 0 91 8 9648 0 1712 1295 88 835 12 71 505 0 6283 9 17 0 74 9 4117 0 1434 1174 103 692 9 59 554 0 3693 6 11 0 84 10 2465 0 861 843 83 480 6 51 486 0 2334 3 7 0 89 11 1009 0 537 2044 1202 734 10 69 652 0 2115 4 7 0 89 12 781 0 455 616 40 418 6 37 407 0 1438 3 4 0 93 13 664 0 472 875 195 648 7 39 452 0 1958 4 5 0 91 14 690 0 403 544 26 380 6 32 393 0 1285 2 4 0 93 15 571 0 469 1151 469 634 6 37 440 0 1824 4 5 0 91 16 717 0 416 486 24 372 7 31 391 0 1165 2 4 0 94 17 467 0 369 413 19 289 6 26 363 0 829 2 3 0 96 18 385 0 385 511 29 512 7 29 402 0 1351 2 3 0 94 19 507 0 323 414 16 368 7 24 361 0 988 2 3 0 95 20 463 0 430 588 47 605 7 29 407 0 1671 3 4 0 93 21 708 0 356 470 29 369 6 24 364 0 1208 2 3 0 95 22 601 0 477 615 35 637 7 29 409 0 1881 3 4 0 93 23 865 0 375 479 28 384 6 25 369 0 1326 2 3 0 94 24 547 0 346 487 20 375 5 34 400 0 1173 2 4 0 94 25 420 0 357 492 61 288 5 30 372 0 854 2 3 0 95 26 360 0 364 540 35 471 5 34 403 0 1241 3 4 0 94 27 426 0 292 441 16 307 5 30 358 0 891 2 3 0 95 28 401 0 365 609 35 583 6 34 416 0 1535 3 4 0 93 29 533 0 331 496 32 345 5 30 375 0 1041 2 4 0 94 30 496 0 480 639 36 614 6 34 422 0 1682 4 4 0 92 31 639 0 336 497 20 364 5 30 368 0 1158 2 4 0 94 CPU minf mjf xcal intr ithr csw icsw migr smtx srw syscl usr sys wt idl 0 2147 0 188 2421 164 338 0 58 88 0 938 1 11 0 88 1 2245 0 144 210 61 418 2 81 148 0 680 2 8 0 90 2 56 0 22 278 81 370 0 35 23 0 423 2 5 0 93 3 2346 0 20 175 24 259 2 58 90 0 829 5 9 0 86 4 1329 0 81 558 125 317 2 29 15 0 2014 1 5 0 93 5 535 0 34 393 60 251 2 47 110 0 361 1 8 0 92 6 302 0 184 670 50 195 0 27 36 0 740 1 9 0 89 7 947 0 128 194 50 200 3 50 195 1 452 2 14 0 84 8 10809 0 203 817 33 266 23 67 72 0 4475 31 15 0 53 9 4898 0 139 2471 88 263 29 57 230 0 4158 41 15 0 44 10 4048 0 149 299 46 147 16 38 139 0 2624 22 11 0 67 11 2291 0 96 1634 357 394 1 75 176 1 1789 4 11 0 85 12 707 0 32 160 62 244 0 41 31 0 448 2 3 0 95 13 47 0 0 483 197 187 1 38 130 1 300 4 6 0 90 14 46 0 2 137 43 273 0 27 19 0 543 2 1 0 97 15 35 0 11 228 109 207 0 33 311 0 254 2 7 0 90 16 0 0 31 2567 106 0 0 0 2007 0 0 0 100 0 0 17 544 0 1672 764 42 254 3 39 135 0 542 2 11 0 87 18 629 0 44 152 26 239 1 53 244 1 988 3 15 0 81 19 3438 0 122 107 22 145 0 31 44 1 1242 3 6 0 91 20 837 0 50 140 34 292 1 52 170 0 901 3 6 0 91 21 1504 0 43 237 19 187 0 18 20 0 1723 4 5 0 91 22 4205 0 100 144 38 336 3 46 39 0 1354 4 10 0 86 23 892 0 55 195 64 251 1 22 26 1 1293 1 3 0 96 24 25 0 0 120 45 141 0 26 34 0 132 1 2 0 97 25 22 0 2 100 23 102 0 22 86 0 77 0 5 0 94 26 0 0 0 73 20 70 0 16 63 0 19 0 2 0 98 27 4 0 6 100 23 150 0 28 82 0 59 0 5 0 95 28 2666 0 6755 130 24 100 7 21 38 0 897 6 11 0 84 29 28 0 1 108 33 146 0 28 129 0 131 0 3 0 96 30 5 0 0 466 30 112 1 25 96 0 45 0 4 0 96 31 9 0 1 99 18 90 0 20 80 0 40 0 5 0 95
and zpool iostat:
root@3d1e2fc2 (storage) ~]$ zpool iostat 1
capacity operations bandwidth
pool alloc free read write read write
---------- ----- ----- ----- ----- ----- -----
zones 75.9T 14.1T 16 713 643K 6.23M
zones 75.9T 14.1T 29 14 199K 83.7K
zones 75.9T 14.1T 1.22K 12 30.5M 120K
zones 75.9T 14.1T 1.28K 17 32.0M 144K
^C
but when I went to run iostat(1M), that command hung, and I couldn't get a shell prompt back with any of CTRL-C, CTRL-Z, or CTRL-/:
[root@3d1e2fc2 (storage) ~]$ iostat 1 ^Z ^C ^C^C^C^Z^Z^Z^Z^Z ^\ ^\
But jobs were hung because zone resets weren't completing. Most of the 112 outstanding resets appeared to be stuck on the "zoneadm halt" step. Eventually we had to send an NMI to restore service.
My stuck iostat turned out to be here:
> ::pgrep iostat | ::walk thread | ::stacks
THREAD STATE SOBJ COUNT
ffffff43a719e760 SLEEP CV 1
swtch+0x141
cv_wait+0x70
ndi_devi_enter+0x7f
devi_config_one+0x3be
ndi_devi_config_one+0xa8
dv_find+0x233
devfs_lookup+0x37
fop_lookup+0xa2
lookuppnvp+0x1f6
prof_lookup_globaldev+0x105
prof_make_names+0x99
prof_filldir+0xe8
sdev_readdir+0xd8
fop_readdir+0x6b
getdents64+0xf5
_sys_sysenter_post_swapgs+0x153
listing /dev inside the zone:
> ffffff43a719e760::findstack -v stack pointer for thread ffffff43a719e760: ffffff01fbd3f4b0 [ ffffff01fbd3f4b0 _resume_from_idle+0xf4() ] ffffff01fbd3f4e0 swtch+0x141() ffffff01fbd3f520 cv_wait+0x70(ffffff42e23136ac, ffffff42e23135c0) ffffff01fbd3f560 ndi_devi_enter+0x7f(ffffff42e2313558, ffffff01fbd3f5f8) ffffff01fbd3f660 devi_config_one+0x3be(ffffff42e2313558, ffffff01fbd3f960, ffffff01fbd3f738, 4000, 0) ffffff01fbd3f6d0 ndi_devi_config_one+0xa8(ffffff42e2313558, ffffff01fbd3f960, ffffff01fbd3f738, 4000) ffffff01fbd3f7b0 dv_find+0x233(ffffff42eca24d18, ffffff01fbd3f960, ffffff01fbd3faa8 , ffffff01fbd3fbc0, ffffff42e2b61840, ffffff42e2691db0, ffffff0100000000) ffffff01fbd3f800 devfs_lookup+0x37(ffffff42ece1be80, ffffff01fbd3f960, ffffff01fbd3faa8, ffffff01fbd3fbc0, 0, ffffff42e2b61840, ffffff42e2691db0, 0, 0, 0 ) ffffff01fbd3f8b0 fop_lookup+0xa2(ffffff42ece1be80, ffffff01fbd3f960, ffffff01fbd3faa8, ffffff01fbd3fbc0, 0, ffffff42e2b61840, ffffff42e2691db0, 0, 0, 0 ) ffffff01fbd3fb10 lookuppnvp+0x1f6(ffffff01fbd3fbc0, 0, 1, 0, ffffff01fbd3fbf0, ffffff42e2b61840, ffffff42e2b61440, ffffff42e2691db0) ffffff01fbd3fc50 prof_lookup_globaldev+0x105(ffffff43c793f398, ffffff42eca23dd8, ffffff43a2274e58, ffffff43a2274e58) ffffff01fbd3fcb0 prof_make_names+0x99(ffffff43c793f398) ffffff01fbd3fcf0 prof_filldir+0xe8(ffffff43c793f398) ffffff01fbd3fd60 sdev_readdir+0xd8(ffffff43cb24d080, ffffff01fbd3fe10, ffffff47e3216120, ffffff01fbd3fe5c, 0, 0) ffffff01fbd3fdf0 fop_readdir+0x6b(ffffff43cb24d080, ffffff01fbd3fe10, ffffff47e3216120, ffffff01fbd3fe5c, 0, 0) ffffff01fbd3feb0 getdents64+0xf5(4, fedf4000, 2000) ffffff01fbd3ff10 _sys_sysenter_post_swapgs+0x153() > ffffff43cb24d080::print vnode_t v_path v_path = 0xffffff43aa0a1210 "/zones/3d1e2fc2-5b9b-4c03-9560-821845899d38/root/dev"
That's blocked on this thread:
> ffffff42e2313558::print struct dev_info devi_flags devi_flags = 0x3 > ffffff42e2313558::print struct dev_info devi_busy_thread devi_busy_thread = 0xffffff01eb66cc40 > 0xffffff01eb66cc40::findstack stack pointer for thread ffffff01eb66cc40: ffffff01eb66c7d0 [ ffffff01eb66c7d0 _resume_from_idle+0xf4() ] ffffff01eb66c800 swtch+0x141() ffffff01eb66c840 cv_wait+0x70() ffffff01eb66c880 taskq_wait+0x43() ffffff01eb66c8c0 kmem_cache_destroy+0x64() ffffff01eb66c8d0 aggr_port_fini+0x10() ffffff01eb66c8f0 aggr_detach+0x48() ffffff01eb66c960 devi_detach+0xa7() ffffff01eb66c9a0 detach_node+0x6c() ffffff01eb66ca10 i_ndi_unconfig_node+0xa2() ffffff01eb66ca40 i_ddi_detachchild+0x33() ffffff01eb66cab0 devi_detach_node+0x81() ffffff01eb66cb40 unconfig_immediate_children+0x18e() ffffff01eb66cbd0 devi_unconfig_common+0x107() ffffff01eb66cc20 mt_config_thread+0x167() ffffff01eb66cc30 thread_start+8()
which is blocked on this taskq:
> 0xffffff01eb66cc40::findstack -v
stack pointer for thread ffffff01eb66cc40: ffffff01eb66c7d0
[ ffffff01eb66c7d0 _resume_from_idle+0xf4() ]
ffffff01eb66c800 swtch+0x141()
ffffff01eb66c840 cv_wait+0x70(ffffff42e2686e72, ffffff42e2686e60)
ffffff01eb66c880 taskq_wait+0x43(ffffff42e2686e40)
ffffff01eb66c8c0 kmem_cache_destroy+0x64(ffffff45c58cc008)
ffffff01eb66c8d0 aggr_port_fini+0x10()
ffffff01eb66c8f0 aggr_detach+0x48(ffffff42e2aeb560, 0)
ffffff01eb66c960 devi_detach+0xa7(ffffff42e2aeb560, 0)
ffffff01eb66c9a0 detach_node+0x6c(ffffff42e2aeb560, 2000)
ffffff01eb66ca10 i_ndi_unconfig_node+0xa2(ffffff42e2aeb560, 4, 2000)
ffffff01eb66ca40 i_ddi_detachchild+0x33(ffffff42e2aeb560, 2000)
ffffff01eb66cab0 devi_detach_node+0x81(ffffff42e2aeb560, 2000)
ffffff01eb66cb40 unconfig_immediate_children+0x18e(ffffff42e2313558, 0, 2000,
ffffffff)
ffffff01eb66cbd0 devi_unconfig_common+0x107(ffffff42e2313558, 0, 2000, ffffffff, 0
)
ffffff01eb66cc20 mt_config_thread+0x167(ffffff44642f99d8)
ffffff01eb66cc30 thread_start+8()
> ffffff42e2686e40::taskq -t
ADDR NAME ACT/THDS Q'ED MAXQ INST
ffffff42e2686e40 kmem_move_taskq 1/ 1 3816702 4858690 0
THREAD STATE SOBJ COUNT
ffffff01e8011c40 ONPROC <NONE> 1
0xf5
kmem_hunt_mag+0x40
kmem_hunt_mags+0x95
kmem_move_buffer+0x1ae
taskq_thread+0x2d0
thread_start+8
For reference, there are 107 "zoneadm halt" processes:
> ::pgrep -x zoneadm | ::print proc_t p_user.u_psargs ! grep -c 'halt"' 107
the other four are booting. They presumably correspond to these stacks:
> ::pgrep -x zoneadm | ::walk thread | ::stacks
THREAD STATE SOBJ COUNT
ffffff42ecf61000 SLEEP SHUTTLE 107
swtch_to+0xb6
shuttle_resume+0x2af
door_call+0x336
doorfs32+0xa7
_sys_sysenter_post_swapgs+0x153
ffffff43bd130760 SLEEP CV 4
swtch+0x141
cv_wait+0x70
ndi_devi_enter+0x7f
devi_config_one+0x3be
ndi_devi_config_one+0xa8
dv_find+0x233
devfs_lookup+0x37
fop_lookup+0xa2
lookuppnvp+0x1f6
lookuppnatcred+0x15e
lookupnameatcred+0xad
lookupnameat+0x39
ldi_vp_from_name+0x12f
ldi_open_by_name+0x6d
ip_create_helper_stream+0x88
udp_ioctl+0xd2
so_ioctl+0x1a9
socket_ioctl+0x27
socket_vop_ioctl+0x29
fop_ioctl+0x55
ioctl+0x9b
_sys_sysenter_post_swapgs+0x153
Let's take a bit of a deeper look at everything:
We have a taskq that things are getting backed up behind:
> ffffff42e2686e40::taskq ADDR NAME ACT/THDS Q'ED MAXQ INST ffffff42e2686e40 kmem_move_taskq 1/ 1 3816702 4858690 0
The reason that we're getting backed up behind them is because of
kmem_cache_destroy() doing a taskq_wait(). That taskq only has one
thread, let's find it:
> ffffff42e2686e40::print taskq_t tq_nthreads
tq_nthreads = 0x1
> ffffff42e2686e40::print taskq_t tq_thr
tq_thr = {
tq_thr._tq_thread = 0xffffff01e8011c40
tq_thr._tq_threadlist = 0xffffff01e8011c40
}
> 0xffffff01e8011c40::findstack -v
stack pointer for thread ffffff01e8011c40: ffffff01e800d860
ffffff01e8011950 0xf5()
ffffff01e8011aa0 kmem_hunt_mag+0x40(ffffff42ed500008, fffffff9fa14f300, 5f, ffffff481427c280, ffffff60974c32e0)
ffffff01e8011b10 kmem_hunt_mags+0x95(ffffff42ed500008, ffffff481427c280)
ffffff01e8011b60 kmem_move_buffer+0x1ae(ffffffe679ce3920)
ffffff01e8011c20 taskq_thread+0x2d0(ffffff42e2686e40)
ffffff01e8011c30 thread_start+8()
> ::cpuinfo ! grep ffffff01e8011c40
28 ffffff430d546580 1f 0 0 60 no no t-3176 ffffff01e8011c40 sched
> ffffff42ed500008::print kmem_cache_t cache_name
cache_name = [ "dnode_t" ]
> ffffff42ed500008::kmem_cache
ADDR NAME FLAG CFLAG BUFSIZE BUFTOTL
ffffff42ed500008 dnode_t 1200 000000 640 22732050
And wow, does that cache have a lot of buffers.
So, let's dig into a bit more and figure out what it means to hunt all the magazines for a given buffer. Let's take a look at what this looks like. Let's start with the number of full magazines:
> ffffff42ed500008::print kmem_cache_t cache_full.ml_list | ::list kmem_magazine_t mag_next ! cat > /files
rmustacc@manta # wc -l files
76379 files
> ffffff42ed500008::print kmem_cache_t cache_magtype->mt_magsize
cache_magtype->mt_magsize = 0x5f
> 0x5f * 0t76379=D
7256005
So, we have over 7 million entries here. We also need to look at all of the per-CPU magazines. However, they have at most 2 * CPUS magazines to worry about, so a drop in the bucket really.
What we don't know is how often we have to do this. This appears to happen each time that a move callback returns KMEM_CBRC_DONT_KNOW. There are a bunch of reasons that might cause that, but it's not clear exactly why that happens. We should be able to put together a D script that proves a bit of this theory. But if so, then we're going to have to better understand this.
Looking at a live system, we were able to gather a bunch of interesting data with DTrace:
[rmustacchi@MS10210 (us-east-1) ~]$ dtrace -n 'fbt::kmem_hunt_mag:entry{ self->t = timestamp; }' -n 'fbt::kmem_hunt_mag:return/self->t/{ @ = quantize(timestamp - self->t); self->t = 0; }' -n 'tick-30s { printa(@); exit(0); }'
dtrace: description 'fbt::kmem_hunt_mag:entry' matched 1 probe
dtrace: description 'fbt::kmem_hunt_mag:return' matched 1 probe
dtrace: description 'tick-30s ' matched 1 probe
^[CPU ID FUNCTION:NAME
1 10924 :tick-30s
value ------------- Distribution ------------- count
256 | 0
512 |@@@@@@@@@@@@@@@@@@@@@@ 7292794
1024 |@@@@@@@@@@@@@@@@@ 5691453
2048 |@ 269332
4096 | 1035
8192 | 1163
16384 | 2773
32768 | 80
65536 | 61
131072 | 9
262144 | 2
524288 | 0
1048576 | 1
2097152 | 0
[rmustacchi@MS10210 (us-east-1) ~]$ dtrace -n 'fbt::dnode_move:return,fbt::zfs_znode_move:return{ @[probefunc,arg1] = count(); }' -n 'tick-10s{ printa(@); exit(0); }'
dtrace: description 'fbt::dnode_move:return,fbt::zfs_znode_move:return' matched 2 probes
dtrace: description 'tick-10s' matched 1 probe
CPU ID FUNCTION:NAME
7 78436 :tick-10s
dnode_move 0 138
dnode_move 4 604
[rmustacchi@MS10210 (us-east-1) ~]$ dtrace -n 'fbt::kmem_hunt_mags:entry{ self->t = timestamp; }' -n 'fbt::kmem_hunt_mags:return/self->t/{ @ = quantize(timestamp - self->t); self->t = 0; }' -n 'tick-30s { printa(@); exit(0); }'
dtrace: description 'fbt::kmem_hunt_mags:entry' matched 1 probe
dtrace: description 'fbt::kmem_hunt_mags:return' matched 1 probe
dtrace: description 'tick-30s ' matched 1 probe
CPU ID FUNCTION:NAME
8 10924 :tick-30s
value ------------- Distribution ------------- count
4096 | 0
8192 | 1
16384 | 0
32768 | 3
65536 | 0
131072 | 4
262144 | 17
524288 |@ 23
1048576 |@@ 55
2097152 |@@ 73
4194304 |@@@@ 139
8388608 |@@@@@@@@@ 308
16777216 |@@@@@@@@@@@@@ 446
33554432 |@@@@@@@@@ 318
67108864 | 0
$ dtrace -n 'fbt::kmem_move_buffer:entry{ self->t = timestamp; }' -n 'fbt::kmem_hunt_mags:return/self->t/{ @ = quantize(timestamp - self->t); self->t = 0; }' -n 'tick-30s { printa(@); exit(0); }' -n 'fbt::dnode_move:return,fbt::zfs_znode_move:return{ @counts[probefunc,arg1] = count(); }'
dtrace: description 'fbt::kmem_move_buffer:entry' matched 1 probe
dtrace: description 'fbt::kmem_hunt_mags:return' matched 1 probe
dtrace: description 'tick-30s ' matched 1 probe
dtrace: description 'fbt::dnode_move:return,fbt::zfs_znode_move:return' matched 2 probes
CPU ID FUNCTION:NAME
8 10924 :tick-30s
value ------------- Distribution ------------- count
32768 | 0
65536 | 2
131072 | 5
262144 | 12
524288 |@ 35
1048576 |@@ 57
2097152 |@@ 79
4194304 |@@@ 123
8388608 |@@@@@@@@@ 318
16777216 |@@@@@@@@@@@@@ 463
33554432 |@@@@@@@@@ 312
67108864 | 0
dnode_move 0 248
dnode_move 4 1407
[rmustacchi@MS10210 (us-east-1) ~]$ pfexec dtrace -n 'fbt::kmem_move_buffer:entry{ self->t = timestamp; }' -n 'fbt::kmem_hunt_mags:return/self->t/{ @ = quantize(timestamp - self->t); self->t = 0; }' -n 'tick-30s { printa(@); exit(0); }' -n 'fbt::dnode_move:return,fbt::zfs_znode_move:return{ @counts[probefunc,arg1] = count(); }'
dtrace: description 'fbt::kmem_move_buffer:entry' matched 1 probe
dtrace: description 'fbt::kmem_hunt_mags:return' matched 1 probe
dtrace: description 'tick-30s ' matched 1 probe
dtrace: description 'fbt::dnode_move:return,fbt::zfs_znode_move:return' matched 2 probes
CPU ID FUNCTION:NAME
8 10924 :tick-30s
value ------------- Distribution ------------- count
16384 | 0
32768 | 1
65536 | 5
131072 | 6
262144 |@ 20
524288 |@ 29
1048576 |@@ 61
2097152 |@@@ 96
4194304 |@@@ 121
8388608 |@@@@@@@@@@ 352
16777216 |@@@@@@@@@@@@ 444
33554432 |@@@@@@@@@ 310
67108864 | 0
dnode_move 0 221
dnode_move 4 1446
[rmustacchi@MS10210 (us-east-1) ~]$ dtrace -q -n 'fbt::kmem_move_buffer:return{ @["proc"] = count(); }' -n 'fbt::kmem_move_begin:entry{ self->t = 1; }' -n 'fbt::kmem_move_begin:return{ self->t = 0; }' -n 'fbt::taskq_dispatch:return/self->t && arg1 == 0/{ @["queue"] = count(); }' -n 'tick-1s{ printa(@); trunc(@); }'
proc 58
proc 50
proc 70
proc 49
proc 56
proc 52
proc 52
proc 54
proc 56
proc 46
proc 65
proc 55
proc 60
proc 59
proc 48
proc 63
proc 59
proc 57
proc 46
proc 56
proc 55
proc 76
proc 72
proc 73
proc 78
proc 65
proc 54
Based on this data we can learn several things. First and foremost is that we seem to be making forward progress, albeit it at a slow rate. We see some latency outliers in terms of how long things are taking, though this may be accounted for by interrupts. Based on this, it seems that the real problem is that we're blocking up everything on work that is progress, but unfortunately scales with memory.
We've had a bunch of different issues with Manta in us-east that I increasingly think are falling out from latency induced by the kmem_move taskq, though the data is still very circumstantial. More investigation is needed.
In the past few weeks, we've seen a bunch of the following events:
- Manta jobs experiencing upload/download failures reported as either timeouts or "socket hang up". We know from experience that this can result from a couple of minutes of idleness on the client socket, which in turn can result from shrimps being unresponsive.
- Marlin jobsupervisors reporting agent timeouts. When we go examine the agents (usually a few minutes later), they seem to be running fine.
- Lackey timeouts resulting in zones becoming disabled.
In all of the cases for which we have data on taskq length and have checked it (which is probably a modest percentage), these events were strongly correlated with kmem_move taskq activity.
I believe this is having significant impact on our ability to complete daily metering, as these spurious agent timeouts trigger retries of multi-hour tasks, causing metering jobs to take a very long time.
We hit this massively again this morning. In the crash dump, we have 233 threads with the same stack, attempting to open a file, blocked on a lock. If we follow the lock holders, we wind up at a thread that's holding a lock (that ultimately hundreds of other threads are blocked on) that's doing a cv_wait() for ARC eviction:
[root@RM08218 (us-east-3) /var/crash/volatile]# mdb 13
mdb: warning: dump is from SunOS 5.11 joyent_20150729T153032Z; dcmds and macros may not match kernel implementation
Loading modules: [ unix genunix specfs dtrace mac cpu.generic uppc apix scsi_vhci ufs ip hook neti sockfs arp usba stmf_sbd stmf zfs mm sd lofs idm mpt_sas crypto random cpc logindmux ptm kvm sppp nsmb smbsrv nfs ipc ]
> ::status
debugging crash dump vmcore.13 (64-bit) from RM08218
operating system: 5.11 joyent_20150729T153032Z (i86pc)
image uuid: (not set)
panic message: BAD TRAP: type=6 (#ud Invalid opcode) rp=ffffff01e80dda00 addr=f
dump content: kernel pages only
> ::stacks -m zfs ! head -80
THREAD STATE SOBJ COUNT
ffffff43ed87c4c0 SLEEP MUTEX 233
swtch+0x141
turnstile_block+0x21a
mutex_vector_enter+0x3a3
zfs_zget+0x51
zfs_root+0x78
fshead_root+0x9e
fsop_root+0x2d
traverse+0x87
lookuppnvp+0x22b
lookuppnatcred+0x15e
lookupnameatcred+0xad
lookupnameat+0x39
vn_openat+0x315
copen+0x1fc
openat64+0x2a
open64+0x25
_sys_sysenter_post_swapgs+0x153
ffffff76d86a3520 SLEEP MUTEX 58
swtch+0x141
turnstile_block+0x21a
mutex_vector_enter+0x3a3
zfs_zget+0x51
zfs_root+0x78
fsop_root+0x2d
traverse+0x87
lookuppnvp+0x22b
lookuppnatcred+0x15e
lookupnameatcred+0xad
lookupnameat+0x39
cstatat_getvp+0x107
cstatat+0x6f
fstatat+0x42
stat+0x22
ffffff4442f797e0 SLEEP CV 35
swtch+0x141
cv_wait+0x70
txg_wait_open+0x83
dmu_tx_wait+0x16f
dmu_tx_assign+0x55
zfs_write+0x4a6
fop_write+0x8b
write+0x250
write32+0x1e
_sys_sysenter_post_swapgs+0x153
ffffff44694fd4e0 SLEEP RWLOCK 8
swtch+0x141
turnstile_block+0x21a
rw_enter_sleep+0x236
dmu_zfetch_find+0x56
dmu_zfetch+0xe0
dbuf_read+0x309
dnode_hold_impl+0xde
dnode_hold+0x28
dmu_read+0x53
zfs_fillpage+0x106
zfs_getpage+0x227
fop_getpage+0x7e
segvn_fault+0xdfa
as_fault+0x37a
pagefault+0x96
trap+0x30c
ffffff45ad4534a0 SLEEP CV 7
swtch+0x141
cv_wait+0x70
txg_wait_synced+0x83
zil_create+0x1c5
zil_commit_writer+0x1a8
zil_commit+0x9e
zfs_fsync+0xba
fop_fsync+0x44
fifo_fsync+0x8d
fifo_inactive+0x90
fop_inactive+0x76
vn_rele+0x82
> ffffff43ed87c4c0::findstack -v
stack pointer for thread ffffff43ed87c4c0: ffffff01eb3493c0
[ ffffff01eb3493c0 _resume_from_idle+0xf4() ]
ffffff01eb3493f0 swtch+0x141()
ffffff01eb349490 turnstile_block+0x21a(ffffff43be94e198, 0, ffffff431e6ffca8,
fffffffffbc07aa0, 0, 0)
ffffff01eb349500 mutex_vector_enter+0x3a3(ffffff431e6ffca8)
ffffff01eb3495a0 zfs_zget+0x51(ffffff431e6ff800, 4, ffffff01eb3495c8)
ffffff01eb349600 zfs_root+0x78(ffffff433a1411d0, ffffff01eb3496b8)
ffffff01eb349660 fshead_root+0x9e(ffffff433a1411d0, ffffff01eb3496b8)
ffffff01eb3496a0 fsop_root+0x2d(ffffff433a1411d0, ffffff01eb3496b8)
ffffff01eb3496f0 traverse+0x87(ffffff01eb3498d8)
ffffff01eb349940 lookuppnvp+0x22b(ffffff01eb349a60, 0, 1, 0, ffffff01eb349c78,
ffffff42e2b70840, ffffff42e2b70840, ffffff61b99444c0)
ffffff01eb3499e0 lookuppnatcred+0x15e(ffffff01eb349a60, 0, 1, 0, ffffff01eb349c78,
0, ffffff61b99444c0)
ffffff01eb349ae0 lookupnameatcred+0xad(8d89d88, 0, 1, 0, ffffff01eb349c78, 0,
ffffff61b99444c0)
ffffff01eb349b30 lookupnameat+0x39(8d89d88, 0, 1, 0, ffffff01eb349c78, 0)
ffffff01eb349ce0 vn_openat+0x315(8d89d88, 0, 2001, 1b6, ffffff01eb349df0, 0,
ffffff0100000012, 0, ffffff010000000c)
ffffff01eb349e50 copen+0x1fc(ffd19553, 8d89d88, 2001, 1b6)
ffffff01eb349e80 openat64+0x2a(ffd19553, 8d89d88, 0, 1b6)
ffffff01eb349eb0 open64+0x25(8d89d88, 0, 1b6)
ffffff01eb349f10 _sys_sysenter_post_swapgs+0x153()
> ffffff431e6ffca8::mutex
ADDR TYPE HELD MINSPL OLDSPL WAITERS
ffffff431e6ffca8 adapt ffffff449dcf6120 - - yes
> ffffff449dcf6120::findstack -v
stack pointer for thread ffffff449dcf6120: ffffff020172c0a0
[ ffffff020172c0a0 _resume_from_idle+0xf4() ]
ffffff020172c0d0 swtch+0x141()
ffffff020172c170 turnstile_block+0x21a(0, 0, ffffff7baacf3c30, fffffffffbc07aa0, 0
, 0)
ffffff020172c1e0 mutex_vector_enter+0x3a3(ffffff7baacf3c30)
ffffff020172c250 dmu_zfetch_find+0x128(ffffffe4a773cab0, ffffff020172c270, 1)
ffffff020172c320 dmu_zfetch+0xe0(ffffffe4a773cab0, 0, 4000, 1)
ffffff020172c390 dbuf_read+0x309(ffffffdf01430130, 0, 2)
ffffff020172c440 dnode_hold_impl+0xde(ffffff58b741a180, 4, 1, fffffffff7d7acc1,
ffffff020172c488)
ffffff020172c470 dnode_hold+0x28(ffffff58b741a180, 4, fffffffff7d7acc1,
ffffff020172c488)
ffffff020172c4d0 dmu_bonus_hold+0x37(ffffff58b741a180, 4, 0, ffffff020172c548)
ffffff020172c500 sa_buf_hold+0x1d(ffffff58b741a180, 4, 0, ffffff020172c548)
ffffff020172c5a0 zfs_zget+0x64(ffffff431e6ff800, 4, ffffff020172c5c8)
ffffff020172c600 zfs_root+0x78(ffffff433a1411d0, ffffff020172c6b8)
ffffff020172c660 fshead_root+0x9e(ffffff433a1411d0, ffffff020172c6b8)
ffffff020172c6a0 fsop_root+0x2d(ffffff433a1411d0, ffffff020172c6b8)
ffffff020172c6f0 traverse+0x87(ffffff020172c8d8)
ffffff020172c940 lookuppnvp+0x22b(ffffff020172ca60, 0, 1, 0, ffffff020172cc78,
ffffff42e2b70840, ffffff42e2b70840, ffffff612163c018)
ffffff020172c9e0 lookuppnatcred+0x15e(ffffff020172ca60, 0, 1, 0, ffffff020172cc78,
0, ffffff612163c018)
ffffff020172cae0 lookupnameatcred+0xad(8d89208, 0, 1, 0, ffffff020172cc78, 0,
ffffff612163c018)
ffffff020172cb30 lookupnameat+0x39(8d89208, 0, 1, 0, ffffff020172cc78, 0)
ffffff020172cce0 vn_openat+0x315(8d89208, 0, 2001, 1b6, ffffff020172cdf0, 0,
ffffff0200000012, 0, ffffff0200000010)
ffffff020172ce50 copen+0x1fc(ffd19553, 8d89208, 2001, 1b6)
ffffff020172ce80 openat64+0x2a(ffd19553, 8d89208, 0, 1b6)
ffffff020172ceb0 open64+0x25(8d89208, 0, 1b6)
ffffff020172cf10 _sys_sysenter_post_swapgs+0x153()
> ffffff7baacf3c30::mutex
ADDR TYPE HELD MINSPL OLDSPL WAITERS
ffffff7baacf3c30 adapt ffffff9a0e2b9b80 - - yes
> ffffff9a0e2b9b80::findstack -v
stack pointer for thread ffffff9a0e2b9b80: ffffff020c1acc80
[ ffffff020c1acc80 _resume_from_idle+0xf4() ]
ffffff020c1accb0 swtch+0x141()
ffffff020c1acd50 turnstile_block+0x21a(ffffff4397233cf0, 0, ffffffffc001dc28,
fffffffffbc07aa0, 0, 0)
ffffff020c1acdc0 mutex_vector_enter+0x3a3(ffffffffc001dc28)
ffffff020c1ace30 dbuf_find+0x62(ffffff58b741a180, 0, 0, 8)
ffffff020c1acfb0 dbuf_prefetch+0xc3(ffffffe4a773c870, 0, 8, 2, 8)
ffffff020c1ad000 dmu_zfetch_fetch+0x5d(ffffffe4a773c870, 8, 1)
ffffff020c1ad040 dmu_zfetch_dofetch+0xbb(ffffffe4a773cab0, ffffff7baacf3c00)
ffffff020c1ad0b0 dmu_zfetch_find+0x3b2(ffffffe4a773cab0, ffffff020c1ad0d0, 1)
ffffff020c1ad180 dmu_zfetch+0xe0(ffffffe4a773cab0, 8000, 4000, 1)
ffffff020c1ad1f0 dbuf_read+0x309(ffffff4636790ef8, 0, 2)
ffffff020c1ad2a0 dnode_hold_impl+0xde(ffffff58b741a180, 59, 1, fffffffff7d7acc1,
ffffff020c1ad2e8)
ffffff020c1ad2d0 dnode_hold+0x28(ffffff58b741a180, 59, fffffffff7d7acc1,
ffffff020c1ad2e8)
ffffff020c1ad330 dmu_bonus_hold+0x37(ffffff58b741a180, 59, 0, ffffff020c1ad3a8)
ffffff020c1ad360 sa_buf_hold+0x1d(ffffff58b741a180, 59, 0, ffffff020c1ad3a8)
ffffff020c1ad400 zfs_zget+0x64(ffffff431e6ff800, 59, ffffff020c1ad530)
ffffff020c1ad4e0 zfs_dirent_lock+0x516(ffffff020c1ad538, ffffff71ee818218,
ffffff020c1ad790, ffffff020c1ad530, 6, 0, 0)
ffffff020c1ad5a0 zfs_dirlook+0x94(ffffff71ee818218, ffffff020c1ad790,
ffffff020c1ad8d8, 0, 0, 0)
ffffff020c1ad640 zfs_lookup+0x3da(ffffff49858d6040, ffffff020c1ad790,
ffffff020c1ad8d8, ffffff020c1ada60, 0, ffffff7887271400, ffffff43c8d76978, 0, 0, 0
)
ffffff020c1ad6f0 fop_lookup+0xa2(ffffff49858d6040, ffffff020c1ad790,
ffffff020c1ad8d8, ffffff020c1ada60, 0, ffffff7887271400, ffffff43c8d76978, 0, 0, 0
)
ffffff020c1ad940 lookuppnvp+0x1fe(ffffff020c1ada60, 0, 1, 0, ffffff020c1adc78,
ffffff7887271400, ffffff7887271400, ffffff43c8d76978)
ffffff020c1ad9e0 lookuppnatcred+0x15e(ffffff020c1ada60, 0, 1, 0, ffffff020c1adc78,
0, ffffff43c8d76978)
ffffff020c1adae0 lookupnameatcred+0xad(80598d9, 0, 1, 0, ffffff020c1adc78, 0,
ffffff43c8d76978)
ffffff020c1adb30 lookupnameat+0x39(80598d9, 0, 1, 0, ffffff020c1adc78, 0)
ffffff020c1adce0 vn_openat+0x315(80598d9, 0, 1, 1b6, ffffff020c1addf0, 0,
ffffff0200000012, 0, ffffff0200000000)
ffffff020c1ade50 copen+0x1fc(ffd19553, 80598d9, 1, 1b6)
ffffff020c1ade80 openat32+0x27(ffd19553, 80598d9, 0, 1b6)
ffffff020c1adeb0 open32+0x25(80598d9, 0, 1b6)
ffffff020c1adf10 _sys_sysenter_post_swapgs+0x153()
> ffffffffc001dc28::mutex
ADDR TYPE HELD MINSPL OLDSPL WAITERS
ffffffffc001dc28 adapt ffffff45ad3fab00 - - yes
> ffffff45ad3fab00::findstack -v
stack pointer for thread ffffff45ad3fab00: ffffff0209210790
[ ffffff0209210790 _resume_from_idle+0xf4() ]
ffffff02092107c0 swtch+0x141()
ffffff0209210860 turnstile_block+0x21a(0, 0, ffffff99e6bc2f40, fffffffffbc07aa0, 0
, 0)
ffffff02092108d0 mutex_vector_enter+0x3a3(ffffff99e6bc2f40)
ffffff0209210940 dbuf_find+0xac(ffffff59fe188040, 1006, 0, 0)
ffffff02092109d0 dbuf_hold_impl+0x6b(ffffff4379e6ed48, 0, 0, 0, 0, fffffffff7d7ac30
, ffffff02092109f8)
ffffff0209210a20 dbuf_hold_level+0x31(ffffff4379e6ed48, 0, 0, fffffffff7d7ac30)
ffffff0209210a50 dbuf_hold+0x21(ffffff4379e6ed48, 0, fffffffff7d7ac30)
ffffff0209210af0 dmu_buf_hold_array_by_dnode+0x121(ffffff4379e6ed48, 0, 2, 1,
fffffffff7d7ac30, ffffff0209210b3c, ffffff0209210b30, ffffff6600000000)
ffffff0209210b90 dmu_read_uio_dnode+0x5a(ffffff4379e6ed48, ffffff0209210de0, 2)
ffffff0209210be0 dmu_read_uio_dbuf+0x48(ffffff445a6d3680, ffffff0209210de0, 2)
ffffff0209210c80 zfs_read+0x1a3(ffffff65e72fe240, ffffff0209210de0, 0,
ffffff66ff690590, 0)
ffffff0209210d20 fop_read+0x8b(ffffff65e72fe240, ffffff0209210de0, 0,
ffffff66ff690590, 0)
ffffff0209210e80 read+0x2a7(44, 92650c0, 2)
ffffff0209210eb0 read32+0x1e(44, 92650c0, 2)
ffffff0209210f10 _sys_sysenter_post_swapgs+0x153()
> ffffff99e6bc2f40::mutex
ADDR TYPE HELD MINSPL OLDSPL WAITERS
ffffff99e6bc2f40 adapt ffffff82208b3820 - - yes
> ffffff82208b3820::findstack -v
stack pointer for thread ffffff82208b3820: ffffff0203b817c0
[ ffffff0203b817c0 _resume_from_idle+0xf4() ]
ffffff0203b817f0 swtch+0x141()
ffffff0203b81890 turnstile_block+0x21a(ffffff56a3dfd0e8, 0, ffffffffc0019bb0,
fffffffffbc07aa0, 0, 0)
ffffff0203b81900 mutex_vector_enter+0x3a3(ffffffffc0019bb0)
ffffff0203b81940 arc_buf_add_ref+0x6a(ffffff939f5679a0, ffffff99e6bc2ee8)
ffffff0203b819d0 dbuf_hold_impl+0xa2(ffffff4379e6ed48, 0, 0, 0, 0, fffffffff7d7ac30
, ffffff0203b819f8)
ffffff0203b81a20 dbuf_hold_level+0x31(ffffff4379e6ed48, 0, 0, fffffffff7d7ac30)
ffffff0203b81a50 dbuf_hold+0x21(ffffff4379e6ed48, 0, fffffffff7d7ac30)
ffffff0203b81af0 dmu_buf_hold_array_by_dnode+0x121(ffffff4379e6ed48, 0, 2, 1,
fffffffff7d7ac30, ffffff0203b81b3c, ffffff0203b81b30, ffffff6600000000)
ffffff0203b81b90 dmu_read_uio_dnode+0x5a(ffffff4379e6ed48, ffffff0203b81de0, 2)
ffffff0203b81be0 dmu_read_uio_dbuf+0x48(ffffff445a6d3680, ffffff0203b81de0, 2)
ffffff0203b81c80 zfs_read+0x1a3(ffffff65e72fe240, ffffff0203b81de0, 0,
ffffff612163c018, 0)
ffffff0203b81d20 fop_read+0x8b(ffffff65e72fe240, ffffff0203b81de0, 0,
ffffff612163c018, 0)
ffffff0203b81e80 read+0x2a7(46, 92670d0, 2)
ffffff0203b81eb0 read32+0x1e(46, 92670d0, 2)
ffffff0203b81f10 _sys_sysenter_post_swapgs+0x153()
> ffffffffc0019bb0::mutex
ADDR TYPE HELD MINSPL OLDSPL WAITERS
ffffffffc0019bb0 adapt ffffff8655614180 - - yes
> ffffff8655614180::findstack -v
stack pointer for thread ffffff8655614180: ffffff01e9f9b3c0
[ ffffff01e9f9b3c0 _resume_from_idle+0xf4() ]
ffffff01e9f9b3f0 swtch+0x141()
ffffff01e9f9b430 cv_wait+0x70(ffffffffc0019318, ffffffffc0019308)
ffffff01e9f9b480 arc_get_data_buf+0x19f(ffffff49f91edf70)
ffffff01e9f9b4c0 arc_buf_clone+0x69(ffffff54ac838148)
ffffff01e9f9b5b0 arc_read+0x904(ffffffdecebff040, ffffff4322e8d000,
ffffff01e9f9b620, fffffffff7cc12c0, ffffff9d4198b780, 2, ffffff0100000180,
ffffff01e9f9b6c8, ffffff01e9f9b6a0)
ffffff01e9f9b730 dbuf_prefetch+0x313(fffffff45618c798, 0, 302, 2, 8)
ffffff01e9f9b780 dmu_zfetch_fetch+0x5d(fffffff45618c798, 2fb, 9)
ffffff01e9f9b7c0 dmu_zfetch_dofetch+0xbb(fffffff45618c9d8, ffffff69233afbf0)
ffffff01e9f9b830 dmu_zfetch_find+0x3b2(fffffff45618c9d8, ffffff01e9f9b850, 20)
ffffff01e9f9b900 dmu_zfetch+0xe0(fffffff45618c9d8, 4100000, 20000, 20)
ffffff01e9f9b970 dbuf_read+0x242(ffffffee25149920, ffffffe76e13b020, 16)
ffffff01e9f9ba10 dmu_buf_hold_array_by_dnode+0x146(fffffff45618c798, 4100000, 5000
, 1, fffffffff7d7ac30, ffffff01e9f9ba5c, ffffff01e9f9ba50, ffffff4500000000)
ffffff01e9f9bab0 dmu_read_uio_dnode+0x5a(fffffff45618c798, ffffff01e9f9bde0, 5000)
ffffff01e9f9bb00 dmu_read_uio_dbuf+0x48(ffffff55c6ff8870, ffffff01e9f9bde0, 5000)
ffffff01e9f9bba0 zfs_read+0x1a3(ffffff498b4d6600, ffffff01e9f9bde0, 0,
ffffff4384b5a720, 0)
ffffff01e9f9bc40 fop_read+0x8b(ffffff498b4d6600, ffffff01e9f9bde0, 0,
ffffff4384b5a720, 0)
ffffff01e9f9bc80 hyprlofs_read+0x34(ffffff54e4ddf900, ffffff01e9f9bde0, 0,
ffffff4384b5a720, 0)
ffffff01e9f9bd20 fop_read+0x8b(ffffff54e4ddf900, ffffff01e9f9bde0, 0,
ffffff4384b5a720, 0)
ffffff01e9f9be80 read+0x2a7(0, 8146000, 8000)
ffffff01e9f9beb0 read32+0x1e(0, 8146000, 8000)
ffffff01e9f9bf10 _sys_sysenter_post_swapgs+0x153()
The ARC reclaim thread is here:
> ::stacks -c arc_reclaim_thread
THREAD STATE SOBJ COUNT
ffffff01e9bfdc40 SLEEP CV 1
swtch+0x141
cv_wait+0x70
taskq_wait+0x43
kmem_cache_reap_now+0x3c
arc_kmem_reap_now+0x53
arc_reclaim_thread+0x21a
thread_start+8
> ffffff01e9bfdc40::findstack -v
stack pointer for thread ffffff01e9bfdc40: ffffff01e9bfdaa0
[ ffffff01e9bfdaa0 _resume_from_idle+0xf4() ]
ffffff01e9bfdad0 swtch+0x141()
ffffff01e9bfdb10 cv_wait+0x70(ffffff42e268ad5a, ffffff42e268ad48)
ffffff01e9bfdb50 taskq_wait+0x43(ffffff42e268ad28)
ffffff01e9bfdb80 kmem_cache_reap_now+0x3c(ffffff42eceec008)
ffffff01e9bfdbb0 arc_kmem_reap_now+0x53()
ffffff01e9bfdc20 arc_reclaim_thread+0x21a()
ffffff01e9bfdc30 thread_start+8()
So it's blocked on that taskq thread, which is here:
> ffffff42e268ad28::taskq -t
ADDR NAME ACT/THDS Q'ED MAXQ INST
ffffff42e268ad28 kmem_taskq 1/ 1 213 768 0
THREAD STATE SOBJ COUNT
ffffff01e8017c40 SLEEP MUTEX 1
swtch+0x141
turnstile_block+0x21a
mutex_vector_enter+0x3a3
kmem_depot_alloc+0x88
kmem_depot_ws_reap+0x70
kmem_cache_reap+0x77
taskq_thread+0x2d0
thread_start+8
Given that the repo lock is also taken by the kmem_move taskq, things are starting to make a little sense. That said, the last link in the chain is missing. That thread is blocked on a mutex, but we can't get the stack for the thread holding it:
> ffffff01e8017c40::findstack -v
stack pointer for thread ffffff01e8017c40: ffffff01e8017960
[ ffffff01e8017960 _resume_from_idle+0xf4() ]
ffffff01e8017990 swtch+0x141()
ffffff01e8017a30 turnstile_block+0x21a(0, 0, ffffff42ed50b1d0, fffffffffbc07aa0, 0
, 0)
ffffff01e8017aa0 mutex_vector_enter+0x3a3(ffffff42ed50b1d0)
ffffff01e8017ae0 kmem_depot_alloc+0x88(ffffff42ed50b008, ffffff42ed50b1e0)
ffffff01e8017b20 kmem_depot_ws_reap+0x70(ffffff42ed50b008)
ffffff01e8017b60 kmem_cache_reap+0x77(ffffff42ed50b008)
ffffff01e8017c20 taskq_thread+0x2d0(ffffff42e268ad28)
ffffff01e8017c30 thread_start+8()
> ffffff42ed50b1d0::mutex
ADDR TYPE HELD MINSPL OLDSPL WAITERS
ffffff42ed50b1d0 adapt ffffff01e8011c40 - - yes
> ffffff01e8011c40::findstack -v
stack pointer for thread ffffff01e8011c40: ffffff01e8011270
ffffff01e8011c40 0xffffff01e8d3ac40()
The depot lock is held, which points to this code in kmem_hunt_mags:
/* Hunt the depot. */
mutex_enter(&cp->cache_depot_lock);
n = cp->cache_magtype->mt_magsize;
for (m = cp->cache_full.ml_list; m != NULL; m = m->mag_next) {
if (kmem_hunt_mag(cp, m, n, buf, tbuf) != NULL) {
mutex_exit(&cp->cache_depot_lock);
return (buf);
}
}
mutex_exit(&cp->cache_depot_lock);
How many full magazines do we have, anyway?
> *dnode_cache::print kmem_cache_t cache_full | ::list kmem_magazine_t mag_next ! wc -l 32847
Yes, you read correctly – there are 32,847 full magazines. Multiply by the magazine size (95) and you have 3.1M buffers that are actually entirely free but can't be freed because the reaping thread can't get the lock because we're too busy trying to defragment a cache by freeing buffers. (!!!)
There are many ways to fix this, but I think the simplest is to have the move thread should basically not consult the magazines at all: a move is kicked by a defrag which itself is kicked by a cache reap; full magazines will be reaped by the reap thread, and buffers that are present in those magazines will be returned to their slabs anyway. Looking at the move stats, on this machine we returned KMEM_CBRC_YES 96.1% of the time and KMEM_CBRC_DONT_KNOW 1.7% of the time; doing invasive, futile work for 1.7% of the cases is fool-hardy – and that's before considering the iteration over 3.1M buffers.
For whatever it's worth, on the 302 dumps in which we had non-zero defrags, here is the percentage of KMEM_CBRC_DONT_KNOW, with output sorted by the number of callbacks:
DUMP CALLBACKS %DONTKNOW
d349015bda171b2acea65aa615f91505 276183824 4.49
320a8c18e0c49818e36635c4d5781204 168332641 3.17
5982fc0918e1288240b45fbcb94b02e2 146940234 4.78
675bdb31abbc8bb7965186893319ebe9 112053071 5.22
66a73fc1530cdc09f97c14aab4d94e52 77218093 4.34
33aad01524eba0f50ab2b64588e8526f 47942462 4.32
3b095192a87588bf71e67d99b1f578c6 29365662 2.53
1620b89dc746f3591e44e2d85543019c 26463768 3.62
9cfc5a874b08a9d1c350da24e59b27d3 21269506 3.55
a1da033680d0a3faaad277b16b5e1a1a 16373257 2.77
e2a2472769e7b818214ae5e2f138a37b 15448841 0.83
f538fc3a9fa68068177a9d0075108d8c 12787953 8.29
3696f4e29c9df9a56ff13353bb4bbba9 12558408 2.55
f1a311f9cbd5f45187c8b882e40a0e24 12437696 4.24
b7c157e1960b82b5e2ff4ce019b385d0 11243756 3.27
af3aaaacbc4dc3cd13f3aa380f16acb0 10864791 13.84
2b65cc2e0dc53ee5d9bb140824dfdf9a 10653476 15.91
17c5298dc62f5587ebd6741d7ce0f00a 10627062 17.36
4a059401d804191d1762186ac3fa5309 10094462 11.25
fd0e8b2651cb5a74614eb631631ff6b7 7690316 0.38
dab916689e68b3f0721bad890e632bc9 7056250 4.18
47a9cde7f15f0ad7dfcdaf8fb30deb0a 6859051 0.44
c85a5b60cbb4add4f4d31de774683155 6435861 1.33
6a837e222e5e4b969bae9a5f6d42183a 6428267 4.20
c3af7251dcfbd7f03b81761bdf7b0752 6377188 1.91
2137d1afcc3f7d379f188a9a04b87ed8 6209598 9.57
28f53b45363776e842a9134e9b542d5e 6174478 4.29
dd7afe8aceaf3d9ad6c8ab580df429d4 5697137 1.24
56b940609ad27f93b2875f1eb5c4582e 5618080 0.35
a5d6ecdf263d3fc84f716d0aee9f7370 5190131 0.36
0c38102577f4214187e8fad3375d7910 3919920 8.88
d287364d8d813b06a88aa5d884e805ae 3517997 3.05
dba6fae8acba5a3fa696ba0ca405a4d9 3120740 4.06
c81a7a50e4447aed770d4cf8110dfcf0 2791764 20.08
2260f5619f1e4898f17df6f1a964c437 2785140 7.92
7f674cc24240c99e3b0468cfc43500fb 2579713 0.90
4056895771e0dbdd19b9ec5902fc85c6 2381808 46.21
af49944a7d785246d93d1a8298c8c568 2253549 6.02
517142bc8fdabce8913b288beaa70784 2236139 9.23
5ae902d321ca7589317856406be0def1 2217412 3.50
ff80de3303b73886ae6fac7891416eaf 2199565 14.57
728d99184eef0631377838bcb5e68c0f 2189680 4.59
e41857a5d584d27830ea5ea9079072c3 2123205 9.21
796991e0f6e98c86feb7347e9b92d090 2113917 2.33
1d20ebaec27b18a5ebe7e24ef9941917 2103142 5.21
15063536f5ffe881c62c478543433f01 2053119 7.64
e96b97e7b653343644e706898d674b49 1907295 2.57
7afc7ba397fa55adea65d9724b686f00 1765535 70.01
58bc62c33399dfbdbb6bb6716446af86 1539337 3.18
6e98e9fb67204e907a7af487380b8630 1533618 3.25
2ade48bfd97d432c8b7da39e9b0cf96d 1500285 2.30
5648ae09e7aea4db0d517a03660a09cf 1476190 27.55
a4da13e1eee343332041d2939be16cb8 1398312 0.38
f2d0a0716785848ac93aa2d9e0a18b64 1317226 2.56
d4611b2ae8bdba4a15e5ecb742c1b62f 1297909 23.02
9439f722d877b81b7d1e0eab29c447c3 1278185 1.68
67e3138bd0f2b6a6d56cf0f5fe9b2c2b 1266340 89.14
c5476e01217acdbecd2886fe6c7bab5d 1149864 0.42
e3522e9f2d16e0d929b83eb828fe5058 1122116 4.63
6bd211e8a40871e914d0499c588049fa 1116914 5.87
2bc211e7238c99dc4b25a9f509e27aa6 1089444 1.92
634ff17a17b32defe3eb7a4d19238522 1062046 1.09
e2a0fa4bb35fc1e214cff09ce62ec298 978204 0.98
16914eaa2f09a300238b04ed71cf65e3 972559 2.94
f1dec9cacb2e282b2f470c15ddc7ebef 965830 3.76
7717d20f34e960d18594dcd96d033641 951935 2.95
27c987c208854fd2cad2a5a3cad2ad7b 915685 2.02
19edd31834b60943a742783025060eab 914643 5.16
f54b4c166222561209b7770c729b08e2 896811 3.03
b757b101bb0c92519125ab62bff54909 884683 1.23
52cefe6966b2fe2e4c3d8e224674288a 878024 2.12
94bf58e51a6db75738f6b6fad3703d26 877734 1.74
dd933c1c57754bf2580fe42c04bd13b8 861923 1.15
2c159a6a17d9bde9e064045f49b5aad5 847944 0.10
cc41a90e066967fe67293cf454694782 797258 9.20
12a5b3edcd5f8785987aef20779d7d32 779537 46.19
b7773ba947d7bc90369038b46a5c01ad 729475 0.79
08bfea3be455022f9cdce432e1799836 721142 1.92
05cc8345b4f1a32d7845e66b90e530f5 711761 6.53
eb0dcad9fb24f4e225741ea98eae875e 695876 1.09
e964c560929bb543f9b87ee9349fc1ac 667077 12.52
b6ab26a1b245f9a9d29188482f187992 659511 7.09
b80c1a837b2faf8ae44b5e77dda9831e 597245 57.17
1bcc39bca886ebb3430ab3e99e3fb748 583637 14.19
28c4e03367c24b9cddd605a3851e0c6e 582447 4.82
6ebceaaf64d3abae27fa1336dd7fb403 569617 1.52
d14afdede9dfacb727d0f11780155211 560861 13.76
7cb15276785d29a5e1cb447c3bcf4e2b 485817 1.81
09b065cd258b0cb8743e570eda2c994a 484606 12.61
ebad5eed8b84dc4f1a4d107395d669d0 481531 4.74
0ea89577ba16992dc74d86d112589e0d 460968 5.21
2539a9a0167535a24f2a07918c3294a2 446319 1.70
12675849fcb76f0eeebd5db87dca28dc 442619 3.77
147b25e20727229fb28add66452b71d2 434441 1.44
36c735d11bbfeffee4d5187d8213b6a9 427376 2.50
c7c9e8ff39564838369f108c5b1df61a 392947 14.88
c5c740f1cd78419e362e413249b30b30 386811 10.17
52e402c09ebbcc9ff87120d21d43edaa 346112 29.72
192ffb93b289822191132d6a4522863b 337730 2.94
85ccd7baba57ea83e6009c90c587fca8 336446 12.77
db56b200e3d22424019fc2415bfcdeb3 328158 32.74
6cd121d4398c168dd30861ae6b2f827f 323312 11.77
80bbd905c8d638b44e01a2a649a4d375 317025 1.03
617e3fbe573626c4ab2577feab1f8e40 314942 2.93
4293ce4aedfa7f70a13b269b4b48021f 305313 2.29
a754657af6298ec0fa61d857d18ede70 302530 1.66
382daf07d730af46ca140b270886af86 297567 4.38
5abb75bd5b220ccd51992b583499e915 283893 1.67
1c9656276f57ba215ef04b3848d157cf 281979 11.49
7c430480c4be364d1061b5c8dfe76ee2 275551 0.54
963e41a8e430bec225dffeddad854bc8 230655 1.32
3e226f111555e9ea6766ff413aa4580f 226435 0.64
1994c984275701f25725fd89e95656c2 223092 6.33
2dabc55e9ae2a28028f329734926fefc 210021 7.01
f4d475043737761927aa3622746c19de c 193139 3.19
5718a7031b2a2fc301ba34e6d82ac5f2 181303 28.75
e1a69f199f11517f6dd08b50cbad09c1 174660 0.16
daca2f4a868acfdcc3a433e5a533d894 157049 1.38
7b41a33ec268004676c41f3cdfda368d 152321 0.66
0adc11edc9f6b7e00ae15e394229fb14 146385 3.74
d56fc2699574aa210cf23479a839b517 143939 6.92
5df31abcaba65483c3477e84bd81a27b 143156 34.25
e95e51af387f0f34aa867baaf4c7812c 143048 6.11
ad1c54e746675d2ee7754426beb21ff3 141196 4.15
79ffe2b444df9ecb98cbce5057f218b7 139633 2.97
40f58b694a1815dd25607f005698f6e3 135483 1.39
18e56102c306041c0a6bc373fdde5980 134396 13.71
e05ce0ff0050cf43ce5c8a0d14c2b9f9 133468 2.89
5dfc850920ea0fb999de7412d1d22a3f 129186 4.48
36213877a483cf69b4688d68e9ba0346 126542 0.89
8c8bb6cdfbc5dc79c8c74b346ca545da 126248 0.39
f7dc51ff5872e8e5e8ccdf95a460cbf9 119762 0.33
0965928ad80c38b7918a4a84c420f722 111950 1.72
b20a6bcd022ce276eb477786e6a2fb9b 102467 2.97
a78d7f920cdf0410bbb5573e45a77916 98046 0.66
a0b124a308064f52466d1f55533e9ca4 96050 2.34
16ee4c6796c70a9033f9d3682f695561 94795 0.75
8f356abd183f92e36f0435df43d59402 94199 2.56
8a0927e6af5b5c2934355a0b2df4f7a7 93435 0.31
75ce9d4e4d2c4686151373073ca52e85 90631 0.83
c58438544b22393786f10a2b6f87e4d5 90346 0.62
5516e397942ea2dde25a28a50a56d83d 89445 1.02
f963d97832f8868c3ba76eb62b3cf2f3 82630 3.78
4848390c464ad577b882aca35ca97109 82218 7.47
4ad4b191fa3493929db292d4125eb870 81959 26.53
827794964e30dcd462582f0a8fd9386f 81174 59.82
54e7933b887771e26afc50dca3f8cef7 79395 1.79
93d70fcda75be4bf02e1d2db56ef4c34 76825 1.64
a288eda3ac0bbe1624e37eebfc8296a3 76750 0.41
d666942527a60b3c451f5822a2cd2f9d 73194 17.72
cb3f0700749e05af398cbeb7f3e3d832 72638 0.63
ddb14a7b16725488c2bc88f465e008f4 71841 0.55
b6cabb3d1259456dec825d38e8dafd0c 71463 0.23
09a6f062f310f483b486f6b448f3a790 64131 0.49
d900d0921bd37624903067b8e282e383 62788 2.27
4f47e5304597137a9d1b54cc6387e6fb 61581 0.99
d84c44bd7b9040c6864392ba72080994 59516 5.73
d091866bc5a8696e5566b6081bc95cc2 59002 1.41
1e543d224f0735fdffa9ba8ce24373a2 58005 12.86
78368e88c2d3602590dc1ed224262b02 52819 0.96
061cd10f29b1029a9a169ced3d4c05d0 52596 0.39
a1469aae93df428a5e25e0c90aa25d1d 49979 2.18
bd90c48b0ba49141e06ecd76d379d7b5 49063 0.62
4de012679309c685bceb7a3c8b1379b3 48882 2.66
4ae326b34da46edc498420ee301cf2de 47411 3.72
ef06807bd47e203aac31997ce49b6bc1 46553 31.22
88eaf4bd1c1cb528588cdb3475ba4668 46090 4.51
e1852ac40ab37cd42bdb275a76d1ba92 46021 6.49
97c4b24d051495d572a4bbf3d03e61c0 45640 4.03
708f69b48013a42e3e72e0694bdd0bba 45440 0.45
21fcc0e6f479f457046d93a19fec0abc 44039 14.40
196067f765ec8b877ed3f7b7bb03013e 42188 5.58
47e17a1fac5fba55d63b3c4fc4b83968 40036 0.37
422e623d91c5846a05815de14afcfdee 35707 1.40
ab36d383c473c6291ef8c84b0c5d3cbe 35198 13.07
09528f87d42419a4bf623288802cc95f 34982 0.40
a3e72020fb344b8b9d94a71aeb32d56b 32243 20.27
1f043226cdf32a7456208a65243c4677 31928 1.78
cf9f34d562fed1b63c32fe3e2a887057 31474 1.46
068f7128e3cbcf8bde52964cb6d86ff4 30927 17.36
b4f68c7971099db65047774134c80a2f 30873 9.35
244461bfe11b6f7c43aacc018959aab7 30835 2.08
07e5ec2dd4a57a735e2c5bef25c150b7 30655 4.50
1dcbd97db3f40874b28dd90aafcc17fb 30617 0.84
7340733a5fcfae49244177abc74a3fea 28625 0.16
df694f9bd93066367f57508f7431f65c 27268 32.39
1b9bdb7b315c84e344d37366557b896b 27256 21.55
7ec67ee8dbea6391e28eb2ed62d85628 24225 0.66
b939b19ccb279ab74e794ef8a0f5748e 22945 0.34
dde1e5c33dbbbcc35a981d6b0059bfd8 21691 2.78
06361bfefc7e2ec2c7a6d136e47c9275 21195 36.71
6bf32500fae459f6c74fc9ab39baa894 19927 0.37
21321fb7c5a924580a0d610000515e09 19812 0.64
68627711e3b7021206edbc4de62cf2c2 18612 84.19
885bf97a5d0297b16298e611469d1cec 18468 0.03
9afb6a10ac410241e329028ee510a6dc 17752 5.78
7e455494504669c6bea531eef0c7d9b7 17223 1.13
0ebb88a740b1da69f7d234f5de0689f8 16958 1.12
b9dc37e5ffeb6266c315f8bee948e8b5 16481 0.29
2f5a121623535338fd026c68a14307aa c 13068 6.06
9d8829b1bb675e601a881e1830f7b3fe 13008 0.66
c6b35ff55f1ceffd4f5f28a1eb994658 12499 0.22
e0904c9b401f7c84330dd35ef5cc33e1 12488 16.00
c628f52e11b54ae0bdf3f477e31ca888 11896 2.09
456ee1f30b86f7c63bd44fc41a34a67a 11460 12.16
f2243c050c252cf4607c0295b31ac5f9 11447 3.04
849e6f93177decde216e5dd925cc73a9 11044 3.59
188f642acdf1e131fb99f00cf52a565a 10499 2.28
02ff7b9cc9fca87adf25239503e8bd92 10438 2.69
a9552973544aa60e4ec72153efe6df9b 10238 9.54
11e6aa49ec320c49ea149e8bb4554961 9566 3.10
4e95de2ccbb0f7d46aef42638c11afdc 9364 3.20
b4ec5232c9a3b3928fc32636126676a3 9081 2.11
55eee843ac19d92ba3254a500dc9dd79 8846 0.61
e3b86ba8f8d6d1151eb96faffd876466 8788 4.35
eaa98455fd2741106e3b9e693614fe4d 8182 8.23
c33f8d5c2e7963c470d13f55e2018b14 7988 1.51
51549b64fabfe0623d8435ee251ed077 6955 1.37
bbcd436ad366ee139205af6a919cf379 6156 0.91
1db5ea3d58098d73a225d3213fca2fec 5763 10.29
f94afdf3e9c95b1c97f1324212e416d1 5651 0.92
8e03ee719208d74fb5d204e33d38c60d 5511 2.09
64d86358b87b18707bae4dfba5e49975 5508 1.65
ddc92a67f2ed6511c72b2d918e0e6f97 5294 2.19
d4ba85da1c16b3d76e44ea1c0c623314 5105 0.86
1f4ecc595b236ef0fa07c5aad08d14e2 4904 1.63
45a970c8dc458bc6b7b98b8eb698a991 4869 5.26
06baf056dae7c720f239e7f3f607643f 4854 0.87
ef516b334b8fb6b4004fd85143e939ad 4733 2.26
0fcfcbc3ea4ce8c3244d41fdc38d7153 4587 48.59
441dca8a871572071dd67ab53d5ba183 4328 1.66
e6a70546af87d8ef341848bf4e4863f6 4210 1.31
dbad7c09387b7c08995408dbcaea127d 4177 10.63
4a58986bda0f492da42d33ea3f9b76e7 4087 15.15
dafa15928dd4696c118c88f872fc3330 3961 1.16
8176f59815d5729f627e84b59d0f4f3e 3960 1.59
cdf5b88d646f5bc457e528488b6251da 3803 0.24
af60eecc1507b1eb6b24ce96392a4ff5 3794 0.47
816bb95555b84c8dae7df3a9bbe1104f 3751 0.56
2e5676ad1605cfca840310cd1cc939a6 3641 3.38
f9bdc18f104d3aa89f12892e408ed3f5 3620 1.93
04b9236661f8fb254b456e65e2069a16 3612 2.85
8de38da49bb9983c68aa9f046c301ec0 3527 0.40
cb20d8f377b33630aa7bc97227d3f999 3453 0.78
2ee01c8007421c64b48ead3109bd2a7b 3446 1.54
66ecafa97cdb4cb32d4f1876b05e52a0 3141 1.27
8b8f934fa46102e78ba748a4e35036a7 2773 30.83
533fdc8d35cc5ed88fdc15ae69f44b93 2490 4.46
065a2ebd18b71d6266a581419569bbfe 2428 0.62
7ecc12f230687f047e948898e9d49d20 2159 1.67
23c80a2f4c1fe51d700bc3cf02199634 2085 2.69
5e0523645ae5d5145d47f80b0717dd5b 2072 10.71
913f3690d9712b55bfef9e3101baad26 1934 1.03
9970072dc2546b0996eebef79ccf981d 1925 10.18
572f8fe3b308fb391a33cdcdeb12bf1b 1832 0.98
dd7bc5ad198f889d825d14459d6d6cab 1729 1.10
da6ba5f37a54705aaf2a1e14fa01272b 1571 4.01
26d780566735c56e630eadebaafe996b 1561 39.78
816a019da081de42e34168af23818e20 1537 1.63
d9243db86ad0bd2e8e12fdc6f0f7c72d 1476 7.45
1e47a72856ecd9bf6a74c6ffeaa39835 1307 1.68
1d1f11c552a23f26f547b190435f6457 1231 38.67
68b4f4bae6c21ac7c9059b8e35487320 1187 2.61
2887c6815bb943948d46aecb237acf38 1150 2.17
1f2058ecb343f3decf0bd1d7aecc2879 1071 2.33
d00610cc6684863be1c43a7d92c93886 1064 33.93
2b9e6d2616f5cdf5272f34ea8eb97e02 1002 20.46
5dfd0ba129bf934ba6bdd030bcead4cc 860 7.09
bcff2cb9411467a59eabb02dfac2863d 801 4.00
c983a739044c2a6fd96555a74c53065e 787 4.70
cc53963d4f74847de6d7fbe0c1722b42 762 1.31
c0a6248f3529dcc05ed8f17d493a89a2 738 0.27
c06916affdb467583731b26d9e0c1c55 728 0.27
65930c21ee3707b990d844032821309d 702 2.42
e4dea78e44542e1fa83cb622cf07c38d 682 0.00
645f44b8c04b762cdde55a2b128bc1d7 673 0.15
749ad6df5ea6ebf6b6497dcb148d4c79 647 3.55
8693b5cb1b5bb13ad349a16702a5be67 637 3.92
9554ecd3905d6bb8ea6693234db33dfd 533 14.26
d79f4eeee7d6f968752d3439ae4c7a11 506 0.59
f04e819d5d905ae616eadc4451e7db31 405 4.20
53ba0d29ab41bdf27b5320b961c929b9 403 0.25
30f417e7c78295bcf45f6f4b899c0f5c 402 4.48
64da01e8a7efb9e0d443ade6f732cd30 362 20.17
d5bd0063dbc1ae7dadee9b01289af004 339 3.54
14c440a88fc03e01f24d22628ebb06af 336 7.74
e5621dee75d6c8c1e56a8035c69afbdc 308 3.90
e4b2a7169bdd52a650a8a29d0ed3733c 306 24.51
2f0a0dc635184f146910224c1f2ac4a2 299 6.35
6230856547207b31376cdb49d551c43e 280 1.07
c62a183eeb40b29919ae5a89f63559ef 226 24.34
94885d274d289277105f57ddd6c371e1 218 10.09
b111db893e23470878a929d2b41e7fbf 190 14.74
9f645b226c8d735a1497312ebd3136e0 147 12.24
7b3ab93aa642d947ec1ba607a056bcc9 119 45.38
73ff09f73bace332f50645ce3e29ce88 72 15.28
079f15401fe71d1f3f494e572123e99e 56 21.43
f80e66528be4fa36e719e9b1645e02a0 30 46.67
d480f15bef5426ebec387a244502cf9b 24 20.83
aadc0a80223becf17e1a400991977bc3 24 4.17
b506552d1450174f7b020c88f77a1228 10 0.00
48d3c93279adc23d749f17ac0bb7d76b 10 0.00
History
Updated by Electric Monk over 2 years ago
Updated by - Status changed from New to Closed
git commit d7db73d165ff4802d277d375e93877471457fed7
commit d7db73d165ff4802d277d375e93877471457fed7
Author: Bryan Cantrill <bryan@joyent.com>
Date: 2018-01-23T21:44:09.000Z
8493 kmem_move taskq appears to be inducing significant system latency
Reviewed by: Robert Mustacchi <rm@joyent.com>
Reviewed by: Dan McDonald <danmcd@joyent.com>
Reviewed by: Paul Dagnelie <pcd@delphix.com>
Approved by: Richard Lowe <richlowe@richlowe.net>