Actions
Bug #1618
closed
zfs causing system to hang in vmem_xalloc()
Start date:
2011-10-09
Due date:
% Done:
100%
Estimated time:
Difficulty:
Hard
Tags:
needs-triage
Gerrit CR:
External Bug:
Description
During the duplication of a zpool to another local zpool using 'zfs send | zfs receive', the system often hangs several hours into the operation. When it's hung, I can no longer ssh into the box although it is pingable. Existing terminal sessions can still run some commands (iostat, vmstat, etc.) but others will just hang such as "mdb -k". The only way to recover is to reboot.
I generated a core dump using kmdb, and initial analysis seems to point to a hang in vmem_xalloc(). The zfs send and receive processes as well as the mdb process that I executed were all hanging in vmem_xalloc(). The thread stacks and symptoms are very similar to the following issues:
http://wesunsolve.net/bugid/id/6989594
http://wesunsolve.net/bugid/id/6942439 (was "fixed" in snv_148 but apparently not for all scenarios)
http://wesunsolve.net/bugid/id/6983481
One of the filesystems in the zpool is a zvol with 8k volblocksize and 150GB allocated space. Usually the system hangs just after this filesystem is duplicated and it moves on to the next one.
I can provide a core dump for analysis. In the meantime I'm trying the workaround in 6983481 of increasing segziosize. System is an HP DL360 G4p with 6GB RAM running oi_151a 64-bit kernel.
Updated by Bryan Leaman over 11 years ago
More details:
# echo '::walk thread | ::findstack -v' | mdb {unix,vmcore}.0 | grep vmem_xalloc
ffffff000c6062c0 vmem_xalloc+0x635(ffffff024aa56000, 20000, 1000, 0, 0, 0, 0,
ffffff000c606580 vmem_xalloc+0x546(ffffff024aa58000, 20000, 1000, 0, 0, 0, 0,
ffffff000c344130 vmem_xalloc+0x635(ffffff024aa56000, 20000, 1000, 0, 0, 0, 0,
ffffff000c3443f0 vmem_xalloc+0x546(ffffff024aa58000, 20000, 1000, 0, 0, 0, 0,
ffffff000cb22e30 vmem_xalloc+0x635(ffffff024aa56000, 20000, 1000, 0, 0, 0, 0,
ffffff000cb230f0 vmem_xalloc+0x546(ffffff024aa58000, 20000, 1000, 0, 0, 0, 0,
# mdb {unix,vmcore}.0
> ::pgrep zfs
S PID PPID PGID SID UID FLAGS ADDR NAME
R 2131 1060 2130 1054 0 0x4a004000 ffffff0260536038 zfs
R 2130 1060 2130 1054 0 0x4a004000 ffffff0258559088 zfs
> ffffff0260536038::walk thread | ::threadlist -v
ADDR PROC LWP CLS PRI WCHAN
ffffff025e5c7500 ffffff0260536038 ffffff025f383c00 1 59 ffffff024aa5601e
PC: _resume_from_idle+0xf1 CMD: zfs receive -vFd data1
stack pointer for thread ffffff025e5c7500: ffffff000c343f90
[ ffffff000c343f90 _resume_from_idle+0xf1() ]
swtch+0x145()
cv_wait+0x61()
vmem_xalloc+0x635()
vmem_alloc+0x161()
segkmem_xalloc+0x90()
segkmem_alloc_vn+0xcd()
segkmem_zio_alloc+0x24()
vmem_xalloc+0x546()
vmem_alloc+0x161()
kmem_slab_create+0x81()
kmem_slab_alloc+0x5b()
kmem_cache_alloc+0x1fa()
zio_data_buf_alloc+0x2c()
arc_get_data_buf+0x18b()
arc_buf_alloc+0xa2()
dbuf_noread+0xa6()
dmu_buf_will_fill+0x1f()
dmu_write+0xef()
restore_write+0xee()
dmu_recv_stream+0x7ab()
zfs_ioc_recv+0x42d()
zfsdev_ioctl+0x15e()
cdev_ioctl+0x45()
spec_ioctl+0x5a()
fop_ioctl+0x7b()
ioctl+0x18e()
_sys_sysenter_post_swapgs+0x149()
> ::pgrep mdb
S PID PPID PGID SID UID FLAGS ADDR NAME
R 2727 2019 2727 2010 0 0x4a004000 ffffff025f43d098 mdb
> ffffff025f43d098::walk thread | ::threadlist -v
ADDR PROC LWP CLS PRI WCHAN
ffffff025549cc00 ffffff025f43d098 ffffff025af020c0 1 60 ffffff024aa5601e
PC: _resume_from_idle+0xf1 CMD: mdb -k
stack pointer for thread ffffff025549cc00: ffffff000cb22c90
[ ffffff000cb22c90 _resume_from_idle+0xf1() ]
swtch+0x145()
cv_wait+0x61()
vmem_xalloc+0x635()
vmem_alloc+0x161()
segkmem_xalloc+0x90()
segkmem_alloc_vn+0xcd()
segkmem_zio_alloc+0x24()
vmem_xalloc+0x546()
vmem_alloc+0x161()
kmem_slab_create+0x81()
kmem_slab_alloc+0x5b()
kmem_cache_alloc+0x1fa()
zio_data_buf_alloc+0x2c()
arc_get_data_buf+0x18b()
arc_buf_alloc+0xa2()
arc_read_nolock+0x12f()
arc_read+0x79()
dsl_read+0x33()
dbuf_read_impl+0x17e()
dbuf_read+0xfd()
dmu_buf_hold_array_by_dnode+0x1c9()
dmu_buf_hold_array+0x6e()
dmu_read_uio+0x4d()
zfs_read+0x255()
fop_read+0x6b()
vn_rdwr+0x17f()
gexec+0x146()
exec_common+0x4e8()
exece+0x1f()
_sys_sysenter_post_swapgs+0x149()
> ::pgrep utmpd
S PID PPID PGID SID UID FLAGS ADDR NAME
R 523 1 523 523 0 0x42000000 ffffff025e45d098 utmpd
> ffffff025e45d098::walk thread | ::threadlist -v
ADDR PROC LWP CLS PRI WCHAN
ffffff0254495480 ffffff025e45d098 ffffff024fed9800 1 60 ffffff024aa5601e
PC: _resume_from_idle+0xf1 CMD: /usr/lib/utmpd
stack pointer for thread ffffff0254495480: ffffff000c606120
[ ffffff000c606120 _resume_from_idle+0xf1() ]
swtch+0x145()
cv_wait+0x61()
vmem_xalloc+0x635()
vmem_alloc+0x161()
segkmem_xalloc+0x90()
segkmem_alloc_vn+0xcd()
segkmem_zio_alloc+0x24()
vmem_xalloc+0x546()
vmem_alloc+0x161()
kmem_slab_create+0x81()
kmem_slab_alloc+0x5b()
kmem_cache_alloc+0x1fa()
zio_data_buf_alloc+0x2c()
arc_get_data_buf+0x18b()
arc_read_nolock+0x409()
arc_read+0x79()
dsl_read+0x33()
dbuf_read_impl+0x17e()
dbuf_read+0xfd()
dmu_buf_hold_array_by_dnode+0x1c9()
dmu_buf_hold_array+0x6e()
dmu_read_uio+0x4d()
zfs_read+0x255()
fop_read+0x6b()
pread+0x22c()
pread32+0x24()
_sys_sysenter_post_swapgs+0x149()
Updated by Bryan Leaman over 11 years ago
Increasing segziosize yields the same panic as reported in 6983481. From my latest core dump:
> ::status
debugging crash dump vmcore.1 (64-bit) from oi151dev
operating system: 5.11 oi_151a (i86pc)
image uuid: 5ec234f3-eeb2-ec51-9908-f077dc971ee1
panic message: hati_pte_map: flags & HAT_LOAD_REMAP
dump content: kernel pages only
> ::panicinfo
cpu 3
thread ffffff000d226c40
message hati_pte_map: flags & HAT_LOAD_REMAP
rdi fffffffffb95c468
rsi ffffff000d225770
rdx 0
rcx 0
r8 ffffff000d2256c0
r9 ffffff000d2257b0
rax 0
rbx 0
rbp ffffff000d2257a0
r10 ffffff024aac1240
r10 ffffff024aac1240
r11 ffffff000d225680
r12 743a6363
r13 ffffff024ad892b8
r14 dfff3573
r15 1
fsbase 0
gsbase ffffff0250dc9580
ds 4b
es 4b
fs 0
gs 1c3
trapno 0
err 0
rip fffffffffb86c870
cs 30
rflags 246
rsp ffffff000d2256b8
ss 38
gdt_hi 0
gdt_lo 200001ef
idt_hi 0
idt_lo 10000fff
ldt 0
task 70
cr0 8005003b
cr2 fed32fe0
cr3 3c00000
cr4 6f8
> $c
vpanic()
hati_pte_map+0x416(ffffff024ad892b8, 0, ffffff0003a3f7e0, 743a6363, 1, 0)
hati_load_common+0x15d(ffffff024aab0e78, ffffff0206a00000, ffffff0003a3f7e0, 427
, 1, 0)
hat_memload+0x81(ffffff024aab0e78, ffffff0206a00000, ffffff0003a3f7e0, 27, 1)
segkmem_xalloc+0x13b(ffffff024aa56000, 0, 20000, 4, 0, fffffffffb898408)
segkmem_alloc_vn+0xcd(ffffff024aa56000, 20000, 4, fffffffffbceeb00)
segkmem_zio_alloc+0x24(ffffff024aa56000, 20000, 4)
vmem_xalloc+0x546(ffffff024aa58000, 20000, 1000, 0, 0, 0)
vmem_alloc+0x161(ffffff024aa58000, 20000, 4)
kmem_slab_create+0x81(ffffff024e6ee348, 4)
kmem_slab_alloc+0x5b(ffffff024e6ee348, 4)
kmem_cache_alloc+0x1fa(ffffff024e6ee348, 4)
zio_data_buf_alloc+0x2c(20000)
arc_get_data_buf+0x18b(ffffff025b4fd430)
arc_buf_alloc+0xa2(ffffff024e69d080, 20000, 0, 0)
arc_read_nolock+0x12f(0, ffffff024e69d080, ffffff0255009180, 0, 0, 6)
arc_read+0x79(0, ffffff024e69d080, ffffff0255009180, ffffff025b4168b0, 0, 0)
dsl_read+0x33(0, ffffff024e69d080, ffffff0255009180, ffffff025b4168b0, 0, 0)
traverse_prefetcher+0x115(ffffff024e69d080, 0, ffffff0255009180,
ffffff025b4168b0, ffffff000d226180, ffffff02511a2a00)
traverse_visitbp+0x149(ffffff000d226b20, ffffff02511a2a00, ffffff025b4168b0,
ffffff0255009180, ffffff000d226180)
traverse_visitbp+0x4dd(ffffff000d226b20, ffffff02511a2a00, ffffff027562eb70,
ffffff0268ac9000, ffffff000d226260)
Updated by Bryan Leaman over 11 years ago
After additional testing and several more hangs, I've noticed some things that may be helpful. Each time the system hangs, the blocked threads all seem to be waiting to allocate virtual memory for zio data buffers and in every crash dump the WCHAN address is always ffffff024aa5601e:
> ffffff024aa5601e::wchaninfo -v
ADDR TYPE NWAITERS THREAD PROC
ffffff024aa5601e cond 14: ffffff02904b5860 fbe
ffffff02d1e37b00 ctfconvert
ffffff02d12f07c0 fbe
ffffff02558860e0 ctfconvert
ffffff02698b8bc0 fbe
ffffff02d2902020 ctfconvert
ffffff0261eaa020 locale
ffffff02559b9160 login
ffffff028ff60860 fbe
ffffff02cf755060 fbe
ffffff028c7873e0 fbe
ffffff02d17ff7c0 fbe
ffffff025588cba0 fbe
ffffff02d28e90e0 fbe
I'm starting to think this is vmem fragmentation, but that's not supposed to be an issue with 64-bit kernels. ::kmem_slabs shows very high waste in the kmem magazines:
> ::kmem_slabs
Partial Unused
Cache Name Slabs Slabs Buffers Buffers Waste
------------------------- -------- -------- --------- --------- ------
kmem_magazine_1 116 116 29116 28144 96.7%
kmem_magazine_3 80 80 10000 9381 93.8%
kmem_magazine_7 140 140 8680 8295 95.6%
kmem_magazine_15 109 109 3379 3127 92.5%
kmem_magazine_31 72 72 1080 950 88.0%
kmem_magazine_47 7 7 70 63 90.0%
kmem_magazine_63 16 16 112 92 82.1%
kmem_magazine_95 12 12 60 47 78.3%
kmem_magazine_143 20 14 60 25 41.7%
In every case, the arc is not completly full and arc_no_grow=0. So I'm not sure where the memory shortage is. Any suggestions would be appreciated.
p = 1047 MB c = 3827 MB c_min = 630 MB c_max = 5043 MB size = 3555 MB arc_no_grow = 0 arc_meta_used = 1178 MB arc_meta_limit = 1260 MB arc_meta_max = 1297 MB
The system is not out of memory:
> ::memstat Page Summary Pages MB %Tot ------------ ---------------- ---------------- ---- Kernel 557030 2175 36% ZFS File Data 611404 2388 39% Anon 41988 164 3% Exec and libs 1663 6 0% Page cache 11396 44 1% Free (cachelist) 15065 58 1% Free (freelist) 314750 1229 20% Total 1553296 6067 Physical 1553295 6067
I'm currently building a debug kernel to see if that identifies any issues. Otherwise I am running out of ideas. The latest hang occurred several hours after the zfs send | zfs recv of a small 5GB snapshot with a 128k recordsize. Until then, the machine was stable for most of the day doing illumos builds.
Updated by Bryan Leaman over 11 years ago
More details as requested by gusev.vitaliy@nexenta.com:
> ::kmastat
cache buf buf buf memory alloc alloc
name size in use total in use succeed fail
------------------------- ------ ------ ------ ---------- --------- -----
kmem_magazine_1 16 917 29116 475136B 4599628 0
kmem_magazine_3 32 474 10000 327680B 222114 0
kmem_magazine_7 64 325 8680 573440B 2088140 0
kmem_magazine_15 128 239 3379 446464B 836697 0
kmem_magazine_31 256 111 1080 294912B 604528 0
kmem_magazine_47 384 0 70 28672B 11708 0
kmem_magazine_63 512 8 112 65536B 1202 0
kmem_magazine_95 768 8 60 49152B 947 0
kmem_magazine_143 1152 26 60 81920B 1997 0
kmem_slab_cache 72 405525 748715 55758848B 10199107 0
kmem_bufctl_cache 24 1164367 1883927 46206976B 11254579 0
kmem_bufctl_audit_cache 192 0 0 0B 0 0
kmem_va_4096 4096 114368 127520 522321920B 223572 0
kmem_va_8192 8192 12848 17504 143392768B 263560 0
kmem_va_12288 12288 209 280 3670016B 8860 0
kmem_va_16384 16384 57339 68400 1120665600B 129429 0
kmem_va_20480 20480 148 156 3407872B 5781 0
kmem_va_24576 24576 70 75 1966080B 2475 0
kmem_va_28672 28672 58 68 2228224B 4014 0
kmem_va_32768 32768 91 124 4063232B 2318 0
kmem_alloc_8 8 209571 236913 1929216B 22702948 0
kmem_alloc_16 16 44484 44929 733184B 1666110 0
kmem_alloc_24 24 89434 103206 2531328B 4151821 0
kmem_alloc_32 32 17734 21000 688128B 44116998 0
kmem_alloc_40 40 16037 25900 1060864B 392790898 0
kmem_alloc_48 48 145207 178865 8826880B 2040065 0
kmem_alloc_56 56 58654 72988 4210688B 1324221 0
kmem_alloc_64 64 60959 100626 6647808B 31835181 0
kmem_alloc_80 80 115716 147050 12046336B 20581335 0
kmem_alloc_96 96 19781 31611 3158016B 1979223 0
kmem_alloc_112 112 23878 30870 3612672B 1404296 0
kmem_alloc_128 128 69688 71579 9457664B 8819142 0
kmem_alloc_160 160 2805 5900 966656B 109602103 0
kmem_alloc_192 192 18955 32320 6619136B 14670871 0
kmem_alloc_224 224 887 4437 1069056B 23685733 0
kmem_alloc_256 256 453 2025 552960B 174021196 0
kmem_alloc_320 320 7185 7344 2506752B 2505081 0
kmem_alloc_384 384 480 560 229376B 38493 0
kmem_alloc_448 448 107 160 81920B 202766 0
kmem_alloc_512 512 476 528 270336B 675589 0
kmem_alloc_640 640 314 360 245760B 203526 0
kmem_alloc_768 768 74 95 77824B 3550 0
kmem_alloc_896 896 5308 8478 7716864B 72981 0
kmem_alloc_1152 1152 190 266 311296B 2984754 0
kmem_alloc_1344 1344 2616 2634 3596288B 5192 0
kmem_alloc_1600 1600 106 135 221184B 87795 0
kmem_alloc_2048 2048 477 552 1130496B 78483415 0
kmem_alloc_2688 2688 146 174 475136B 135988 0
kmem_alloc_4096 4096 205 219 897024B 490695 0
kmem_alloc_8192 8192 125 131 1073152B 263653 0
kmem_alloc_12288 12288 28 34 417792B 1626 0
kmem_alloc_16384 16384 62 67 1097728B 1110 0
kmem_alloc_24576 24576 48 52 1277952B 743 0
kmem_alloc_32768 32768 74 80 2621440B 17730 0
kmem_alloc_40960 40960 15 21 860160B 136 0
kmem_alloc_49152 49152 8 13 638976B 73 0
kmem_alloc_57344 57344 8 14 802816B 77 0
kmem_alloc_65536 65536 11 15 983040B 68 0
kmem_alloc_73728 73728 4 9 663552B 70 0
kmem_alloc_81920 81920 1 6 491520B 57 0
kmem_alloc_90112 90112 2 4 360448B 9 0
kmem_alloc_98304 98304 3 6 589824B 16 0
kmem_alloc_106496 106496 5 7 745472B 9 0
kmem_alloc_114688 114688 5 5 573440B 5 0
kmem_alloc_122880 122880 4 4 491520B 4 0
kmem_alloc_131072 131072 9 11 1441792B 15 0
streams_mblk 64 52382 52576 3473408B 124087963 0
streams_dblk_16 128 167 372 49152B 170530 0
streams_dblk_80 192 288 520 106496B 5051344 0
streams_dblk_144 256 0 150 40960B 183045108 0
streams_dblk_208 320 238 372 126976B 79036 0
streams_dblk_272 384 46408 46420 19013632B 12880159 0
streams_dblk_336 448 4902 4920 2519040B 22424719 0
streams_dblk_528 640 1 174 118784B 2021907 0
streams_dblk_1040 1152 0 21 24576B 49114 0
streams_dblk_1488 1600 0 40 65536B 11506 0
streams_dblk_1936 2048 0 18 36864B 8869135 0
streams_dblk_2576 2688 1 18 49152B 130 0
streams_dblk_3856 3968 0 5 20480B 28014049 0
streams_dblk_8192 112 0 31 4096B 17414618 0
streams_dblk_12048 12160 0 6 73728B 2859151 0
streams_dblk_16384 112 0 31 4096B 5004 0
streams_dblk_20240 20352 0 4 81920B 1631 0
streams_dblk_24576 112 0 31 4096B 1410 0
streams_dblk_28432 28544 0 4 114688B 1502 0
streams_dblk_32768 112 0 62 8192B 5543552 0
streams_dblk_36624 36736 0 0 0B 0 0
streams_dblk_40960 112 0 0 0B 0 0
streams_dblk_44816 44928 0 0 0B 0 0
streams_dblk_49152 112 0 0 0B 0 0
streams_dblk_53008 53120 0 0 0B 0 0
streams_dblk_57344 112 0 0 0B 0 0
streams_dblk_61200 61312 0 0 0B 0 0
streams_dblk_65536 112 0 0 0B 0 0
streams_dblk_69392 69504 0 0 0B 0 0
streams_dblk_73728 112 0 0 0B 0 0
streams_dblk_esb 112 0 0 0B 0 0
streams_fthdr 408 0 0 0B 0 0
streams_ftblk 376 0 0 0B 0 0
multidata 248 0 0 0B 0 0
multidata_pdslab 7112 0 0 0B 0 0
multidata_pattbl 32 0 0 0B 0 0
log_cons_cache 48 9 83 4096B 97 0
taskq_ent_cache 56 2423 3408 196608B 92673 0
taskq_cache 280 139 154 45056B 239 0
kmem_io_64G_128 128 0 31 4096B 1683 0
kmem_io_64G_256 256 0 30 8192B 24 0
kmem_io_64G_512 512 0 16 8192B 33 0
kmem_io_64G_1024 1024 0 0 0B 0 0
kmem_io_64G_2048 2048 5250 5250 10752000B 5250 0
kmem_io_64G_4096 4096 0 0 0B 0 0
kmem_io_4G_128 128 2 31 4096B 2 0
kmem_io_4G_256 256 1 15 4096B 105 0
kmem_io_4G_512 512 26 32 16384B 26 0
kmem_io_4G_1024 1024 3 16 16384B 945 0
kmem_io_4G_2048 2048 0 8 16384B 155 0
kmem_io_4G_4096 4096 1023 1023 4190208B 1023 0
kmem_io_2G_128 128 1 31 4096B 1 0
kmem_io_2G_256 256 0 0 0B 0 0
kmem_io_2G_512 512 0 0 0B 0 0
kmem_io_2G_1024 1024 0 0 0B 0 0
kmem_io_2G_2048 2048 0 0 0B 0 0
kmem_io_2G_4096 4096 2 4 16384B 54 0
kmem_io_16M_128 128 0 0 0B 0 0
kmem_io_16M_256 256 0 0 0B 0 0
kmem_io_16M_512 512 0 0 0B 0 0
kmem_io_16M_1024 1024 0 0 0B 0 0
kmem_io_16M_2048 2048 0 0 0B 0 0
kmem_io_16M_4096 4096 0 0 0B 0 0
id32_cache 32 0 0 0B 0 0
bp_map_4096 4096 0 0 0B 0 0
bp_map_8192 8192 0 16 131072B 1307 0
bp_map_12288 12288 0 10 131072B 1411 0
bp_map_16384 16384 0 8 131072B 1478 0
bp_map_20480 20480 0 6 131072B 1511 0
bp_map_24576 24576 0 10 262144B 1425 0
bp_map_28672 28672 0 8 262144B 1459 0
bp_map_32768 32768 0 12 393216B 6293728 0
htable_t 72 4133 4510 335872B 1249104 0
hment_t 64 49000 72292 4775936B 60650543 0
hat_t 176 116 176 32768B 166519 0
HatHash 8192 2 8 65536B 1919 0
HatVlpHash 4096 114 117 479232B 184807 0
zfs_file_data_4096 4096 1732 1920 7864320B 1364826 0
zfs_file_data_8192 8192 268345 347328 2845310976B 11958066 0
zfs_file_data_12288 12288 977 980 12845056B 13104 0
zfs_file_data_16384 16384 156 160 2621440B 4665 0
zfs_file_data_20480 20480 786 786 17170432B 15196 0
zfs_file_data_24576 24576 210 210 5505024B 4807 0
zfs_file_data_28672 28672 378 380 12451840B 8164 0
zfs_file_data_32768 32768 108 108 3538944B 3574 0
segkp_4096 4096 111 160 655360B 117669 0
segkp_8192 8192 0 0 0B 0 0
segkp_12288 12288 0 0 0B 0 0
segkp_16384 16384 0 0 0B 0 0
segkp_20480 20480 0 0 0B 0 0
umem_np_4096 4096 0 32 131072B 143 0
umem_np_8192 8192 0 16 131072B 9 0
umem_np_12288 12288 0 0 0B 0 0
umem_np_16384 16384 0 8 131072B 84 0
umem_np_20480 20480 0 6 131072B 15 0
umem_np_24576 24576 0 5 131072B 3 0
umem_np_28672 28672 0 0 0B 0 0
umem_np_32768 32768 0 4 131072B 9 0
mod_hash_entries 24 280 501 12288B 2706 0
ipp_mod 304 0 0 0B 0 0
ipp_action 368 0 0 0B 0 0
ipp_packet 64 0 0 0B 0 0
seg_cache 96 4796 6232 622592B 6690154 0
seg_pcache 104 14 76 8192B 60 0
fnode_cache 176 5 20 4096B 41 0
pipe_cache 320 24 48 16384B 14178 0
snode_cache 152 239 312 49152B 262637 0
dv_node_cache 176 86 220 40960B 1658 0
mac_impl_cache 13568 1 3 40960B 2 0
mac_ring_cache 192 4 20 4096B 8 0
flow_entry_cache 27112 3 8 217088B 6 0
flow_tab_cache 216 1 18 4096B 2 0
mac_soft_ring_cache 376 6 10 4096B 12 0
mac_srs_cache 3240 3 5 16384B 6 0
mac_bcast_grp_cache 80 2 50 4096B 4 0
mac_client_impl_cache 1328 1 3 4096B 2 0
mac_promisc_impl_cache 112 0 0 0B 0 0
dls_link_cache 344 1 11 4096B 2 0
dls_devnet_cache 360 1 11 4096B 2 0
sdev_node_cache 256 1551 1560 425984B 1658 0
dev_info_node_cache 680 181 240 163840B 876 0
ndi_fm_entry_cache 32 288 500 16384B 624 0
thread_cache 912 368 504 516096B 621086 0
lwp_cache 1760 792 837 1523712B 127844 0
turnstile_cache 64 1088 1488 98304B 538059 0
tslabel_cache 48 2 83 4096B 2 0
cred_cache 184 94 147 28672B 51549 0
rctl_cache 48 1689 2241 110592B 1056083 0
rctl_val_cache 64 3364 4464 294912B 2270889 0
task_cache 160 41 75 12288B 147 0
kmem_defrag_cache 216 2 18 4096B 2 0
kmem_move_cache 56 0 142 8192B 56646 0
rootnex_dmahdl 2592 6994 7086 19349504B 78414192 0
timeout_request 128 2 31 4096B 3 0
cyclic_id_cache 72 22 55 4096B 28 0
callout_cachebabecafe 80 501 527 69632B 501 0
callout_lcachebabecafe 48 874 930 61440B 874 0
dnlc_space_cache 24 0 0 0B 0 0
vfs_cache 208 53 95 20480B 6192 0
vn_cache 208 69611 125400 34242560B 933980 0
vsk_anchor_cache 40 21 100 4096B 36 0
file_cache 56 462 639 36864B 2431053 0
stream_head_cache 376 139 170 69632B 41489 0
queue_cache 656 281 300 204800B 63067 0
syncq_cache 160 11 25 4096B 31 0
qband_cache 64 0 0 0B 0 0
linkinfo_cache 48 7 83 4096B 9 0
ciputctrl_cache 256 0 0 0B 0 0
serializer_cache 64 27 62 4096B 56 0
as_cache 232 115 187 45056B 166518 0
marker_cache 120 0 33 4096B 4126 0
anon_cache 48 48027 82834 4087808B 15986891 0
anonmap_cache 112 3438 4760 557056B 3375306 0
segvn_cache 168 4796 6279 1118208B 5013234 0
segvn_szc_cache1 4096 0 0 0B 0 0
flk_edges 48 0 0 0B 0 0
fdb_cache 104 0 0 0B 0 0
timer_cache 136 1 29 4096B 1 0
vmu_bound_cache 56 0 0 0B 0 0
vmu_object_cache 64 0 0 0B 0 0
physio_buf_cache 248 0 0 0B 0 0
process_cache 3880 121 125 512000B 97691 0
kcf_sreq_cache 56 0 0 0B 0 0
kcf_areq_cache 296 0 0 0B 0 0
kcf_context_cache 112 0 31 4096B 11248 0
clnt_clts_endpnt_cache 88 0 0 0B 0 0
zio_cache 856 13 1107 1007616B 576590051 0
zio_link_cache 48 0 3320 163840B 622661688 0
zio_buf_512 512 86480 118672 60760064B 33898508 0
zio_data_buf_512 512 995 1952 999424B 130223 0
zio_buf_1024 1024 2626 2876 2945024B 27507063 0
zio_data_buf_1024 1024 673 1056 1081344B 68046 0
zio_buf_1536 1536 550 624 958464B 47635687 0
zio_data_buf_1536 1536 2337 2600 3993600B 56112 0
zio_buf_2048 2048 297 322 659456B 38513745 0
zio_data_buf_2048 2048 1636 1642 3362816B 39457 0
zio_buf_2560 2560 152 176 450560B 25778624 0
zio_data_buf_2560 2560 1108 1112 2846720B 29162 0
zio_buf_3072 3072 111 128 393216B 19144907 0
zio_data_buf_3072 3072 641 648 1990656B 24087 0
zio_buf_3584 3584 74 96 344064B 12761791 0
zio_data_buf_3584 3584 471 480 1720320B 31594 0
zio_buf_4096 4096 1826 1831 7499776B 9181311 0
zio_data_buf_4096 4096 401 403 1650688B 1369951 0
zio_buf_5120 5120 79 96 491520B 7889481 0
zio_data_buf_5120 5120 642 644 3297280B 38339 0
zio_buf_6144 6144 55 70 430080B 5713191 0
zio_data_buf_6144 6144 419 420 2580480B 27961 0
zio_buf_7168 7168 49 56 401408B 1595006 0
zio_data_buf_7168 7168 299 304 2179072B 19621 0
zio_buf_8192 8192 18 48 393216B 24749683 0
zio_data_buf_8192 8192 268295 268345 2198282240B 35581653 0
zio_buf_10240 10240 26 34 348160B 444648 0
zio_data_buf_10240 10240 420 420 4300800B 31172 0
zio_buf_12288 12288 17 21 258048B 385183 0
zio_data_buf_12288 12288 279 280 3440640B 20533 0
zio_buf_14336 14336 16 24 344064B 260375 0
zio_data_buf_14336 14336 258 260 3727360B 15297 0
zio_buf_16384 16384 57156 57177 936787968B 9347304 0
zio_data_buf_16384 16384 152 156 2555904B 13260 0
zio_buf_20480 20480 77 81 1658880B 531087 0
zio_data_buf_20480 20480 276 276 5652480B 21820 0
zio_buf_24576 24576 12 18 442368B 460786 0
zio_data_buf_24576 24576 210 210 5160960B 15773 0
zio_buf_28672 28672 11 16 458752B 388570 0
zio_data_buf_28672 28672 112 112 3211264B 11596 0
zio_buf_32768 32768 6 11 360448B 446748 0
zio_data_buf_32768 32768 107 108 3538944B 13006 0
zio_buf_36864 36864 23 28 1032192B 235438 0
zio_data_buf_36864 36864 76 78 2875392B 8573 0
zio_buf_40960 40960 1 5 204800B 209746 0
zio_data_buf_40960 40960 61 63 2580480B 6769 0
zio_buf_45056 45056 2 8 360448B 170825 0
zio_data_buf_45056 45056 47 49 2207744B 5656 0
zio_buf_49152 49152 3 8 393216B 156767 0
zio_data_buf_49152 49152 47 50 2457600B 5080 0
zio_buf_53248 53248 10 15 798720B 134202 0
zio_data_buf_53248 53248 38 41 2183168B 4856 0
zio_buf_57344 57344 1 4 229376B 118866 0
zio_data_buf_57344 57344 56 58 3325952B 4287 0
zio_buf_61440 61440 0 4 245760B 98056 0
zio_data_buf_61440 61440 64 66 4055040B 3909 0
zio_buf_65536 65536 0 5 327680B 1521718 0
zio_data_buf_65536 65536 36 39 2555904B 2959 0
zio_buf_69632 69632 0 5 348160B 92961 0
zio_data_buf_69632 69632 28 29 2019328B 2907 0
zio_buf_73728 73728 0 5 368640B 85055 0
zio_data_buf_73728 73728 27 29 2138112B 2586 0
zio_buf_77824 77824 3 8 622592B 84550 0
zio_data_buf_77824 77824 25 27 2101248B 2583 0
zio_buf_81920 81920 0 5 409600B 86278 0
zio_data_buf_81920 81920 7 11 901120B 2231 0
zio_buf_86016 86016 1 7 602112B 84425 0
zio_data_buf_86016 86016 20 22 1892352B 2111 0
zio_buf_90112 90112 1 7 630784B 76066 0
zio_data_buf_90112 90112 16 18 1622016B 1696 0
zio_buf_94208 94208 0 5 471040B 74458 0
zio_data_buf_94208 94208 13 16 1507328B 1711 0
zio_buf_98304 98304 0 5 491520B 79836 0
zio_data_buf_98304 98304 12 14 1376256B 1335 0
zio_buf_102400 102400 0 5 512000B 96068 0
zio_data_buf_102400 102400 15 18 1843200B 1223 0
zio_buf_106496 106496 0 4 425984B 80083 0
zio_data_buf_106496 106496 6 9 958464B 1182 0
zio_buf_110592 110592 0 5 552960B 73802 0
zio_data_buf_110592 110592 12 15 1658880B 1339 0
zio_buf_114688 114688 0 6 688128B 69899 0
zio_data_buf_114688 114688 3 8 917504B 1060 0
zio_buf_118784 118784 0 3 356352B 64731 0
zio_data_buf_118784 118784 4 7 831488B 1221 0
zio_buf_122880 122880 0 5 614400B 68852 0
zio_data_buf_122880 122880 2 9 1105920B 1017 0
zio_buf_126976 126976 0 5 634880B 93213 0
zio_data_buf_126976 126976 3 7 888832B 903 0
zio_buf_131072 131072 285 290 38010880B 1378932 0
zio_data_buf_131072 131072 1560 1562 204734464B 291569 0
sa_cache 56 65154 119919 6918144B 370950 0
dnode_t 744 99992 100001 74473472B 404068 0
dmu_buf_impl_t 192 110729 153740 31485952B 25952186 0
arc_buf_hdr_t 176 561027 828256 154206208B 121785562 0
arc_buf_t 56 355682 757002 43671552B 116102653 0
zil_lwb_cache 192 3 80 16384B 443 0
zfs_znode_cache 248 65154 65232 16699392B 429429 0
audit_proc 40 120 300 12288B 85933 0
drv_secobj_cache 296 0 0 0B 0 0
dld_str_cache 304 2 13 4096B 4 0
ip_minor_arena_sa_1 1 21 128 128B 7814 0
ip_minor_arena_la_1 1 3 64 64B 4323 0
ip_conn_cache 720 0 5 4096B 2 0
tcp_conn_cache 1808 46 60 122880B 2042 0
udp_conn_cache 1256 20 39 53248B 9052 0
rawip_conn_cache 1096 0 0 0B 0 0
rts_conn_cache 816 5 9 8192B 11 0
ire_cache 352 29 33 12288B 146 0
ncec_cache 200 16 100 20480B 249 0
nce_cache 88 16 135 12288B 385 0
rt_entry 152 21 52 8192B 127 0
radix_mask 32 3 125 4096B 5 0
radix_node 120 2 33 4096B 2 0
ipsec_actions 72 0 0 0B 0 0
ipsec_selectors 80 0 0 0B 0 0
ipsec_policy 80 0 0 0B 0 0
tcp_timercache 88 95 315 28672B 7943 0
tcp_notsack_blk_cache 24 0 167 4096B 791 0
squeue_cache 168 6 20 4096B 6 0
sctp_conn_cache 2528 0 0 0B 0 0
sctp_faddr_cache 176 0 0 0B 0 0
sctp_set_cache 24 0 0 0B 0 0
sctp_ftsn_set_cache 16 0 0 0B 0 0
dce_cache 152 8 26 4096B 17 0
ire_gw_secattr_cache 24 0 0 0B 0 0
socket_cache 640 45 66 45056B 9500 0
socktpi_cache 944 0 4 4096B 1 0
socktpi_unix_cache 944 22 28 28672B 41 0
sock_sod_cache 648 0 0 0B 0 0
exacct_object_cache 40 0 0 0B 0 0
kssl_cache 1624 0 0 0B 0 0
fctl_cache 112 0 0 0B 0 0
tl_cache 432 40 45 20480B 138 0
keysock_1 1 0 0 0B 0 0
spdsock_1 1 0 64 64B 1 0
namefs_inodes_1 1 27 128 128B 6142 0
port_cache 80 5 50 4096B 5 0
softmac_cache 568 1 7 4096B 2 0
softmac_upper_cache 232 0 0 0B 0 0
Hex0xffffff024e6a4558_minor_1 1 0 0 0B 0 0
Hex0xffffff024e6a4560_minor_1 1 0 0 0B 0 0
lnode_cache 32 2 125 4096B 960 0
idm_buf_cache 240 0 0 0B 0 0
idm_task_cache 2432 0 0 0B 0 0
idm_tx_pdu_cache 464 0 0 0B 0 0
idm_rx_pdu_cache 513 0 0 0B 0 0
idm_128k_buf_cache 131072 0 0 0B 0 0
authkern_cache 72 0 55 4096B 5453453 0
authnone_cache 72 0 0 0B 0 0
authloopback_cache 72 0 0 0B 0 0
authdes_cache_handle 80 0 0 0B 0 0
dtrace_state_cache 16384 0 1 16384B 3 0
ufs_inode_cache 368 0 0 0B 0 0
directio_buf_cache 272 0 0 0B 0 0
lufs_save 24 0 0 0B 0 0
lufs_bufs 256 0 0 0B 0 0
lufs_mapentry_cache 112 0 0 0B 0 0
pty_map 64 2 62 4096B 12 0
sppptun_map 440 0 0 0B 0 0
smb_share_cache 136 4 29 4096B 4 0
smb_unexport_cache 272 0 0 0B 0 0
smb_vfs_cache 48 2 83 4096B 2 0
smb_mbc_cache 56 0 0 0B 0 0
rnode_cache 656 0 6 4096B 3 0
nfs_access_cache 56 0 71 4096B 1 0
client_handle_cache 32 0 125 4096B 17 0
rnode4_cache 968 0 0 0B 0 0
svnode_cache 40 0 0 0B 0 0
nfs4_access_cache 56 0 0 0B 0 0
client_handle4_cache 32 0 0 0B 0 0
nfs4_ace4vals_cache 48 0 0 0B 0 0
nfs4_ace4_list_cache 264 0 0 0B 0 0
NFS_idmap_cache 48 0 0 0B 0 0
smb_node_cache 752 1 20 16384B 137 0
smb_oplock_break_cache 32 0 0 0B 0 0
smb_txreq 66592 0 7 466944B 772 0
smb_dtor_cache 40 0 100 4096B 210 0
smb_request_cache 1608 0 10 16384B 794 0
smb_session_cache 1744 1 14 24576B 99 0
smb_user_cache 288 0 13 4096B 91 0
smb_tree_cache 1752 0 7 12288B 52 0
smb_ofile_cache 480 0 8 4096B 67 0
smb_odir_cache 8600 0 0 0B 0 0
smb_opipe_cache 216 0 18 4096B 66 0
smb_event_cache 72 3 55 4096B 297 0
crypto_session_cache 104 0 0 0B 0 0
lm_vnode 184 0 0 0B 0 0
lm_xprt 32 0 0 0B 0 0
lm_sysid 160 0 0 0B 0 0
lm_client 128 0 0 0B 0 0
lm_async 32 0 0 0B 0 0
lm_sleep 96 0 0 0B 0 0
lm_config 80 2 50 4096B 2 0
------------------------- ------ ------ ------ ---------- --------- -----
Total [hat_memload] 5111808B 61899647 0
Total [kmem_msb] 104321024B 29877295 0
Total [kmem_va] 1801715712B 640009 0
Total [kmem_default] 1583808512B 3214471011 0
Total [kmem_io_64G] 10772480B 6990 0
Total [kmem_io_4G] 4247552B 2256 0
Total [kmem_io_2G] 20480B 55 0
Total [bp_map] 1441792B 6302319 0
Total [umem_np] 786432B 263 0
Total [zfs_file_data] 2907308032B 13372402 0
Total [zfs_file_data_buf] 2504310784B 37917426 0
Total [segkp] 655360B 117669 0
Total [ip_minor_arena_sa] 128B 7814 0
Total [ip_minor_arena_la] 64B 4323 0
Total [spdsock] 64B 1 0
Total [namefs_inodes] 128B 6142 0
------------------------- ------ ------ ------ ---------- --------- -----
vmem memory memory memory alloc alloc
name in use total import succeed fail
------------------------- ---------- ----------- ---------- --------- -----
heap 2356097024B 1088595951616B 0B 53642 0
vmem_metadata 125435904B 125435904B 125435904B 29483 0
vmem_seg 120279040B 120279040B 120279040B 29350 0
vmem_hash 4904448B 4911104B 4911104B 84 0
vmem_vmem 254800B 282160B 245760B 80 0
static 0B 0B 0B 0 0
static_alloc 0B 0B 0B 0 0
hat_memload 5111808B 5111808B 5111808B 6500 0
kstat 453128B 471040B 405504B 1494 0
kmem_metadata 111153152B 134873088B 134873088B 47895 0
kmem_msb 104321024B 104321024B 104321024B 47551 0
kmem_cache 350200B 438272B 438272B 481 0
kmem_hash 6373376B 6393856B 6393856B 786 0
kmem_log 131808B 135168B 135168B 6 0
kmem_firewall_va 99987456B 99987456B 99987456B 5848 0
kmem_firewall 0B 0B 0B 0 0
kmem_oversize 99895134B 99987456B 99987456B 5848 0
mod_sysfile 265B 4096B 4096B 7 0
kmem_va 1860415488B 1860415488B 1860415488B 33954 0
kmem_default 1583808512B 1583808512B 1583808512B 465612 0
kmem_io_64G 10772480B 10772480B 10772480B 2631 0
kmem_io_4G 4247552B 4247552B 4247552B 1045 0
kmem_io_2G 20480B 20480B 20480B 6 0
kmem_io_16M 0B 0B 0B 0 0
bp_map 1441792B 1441792B 1441792B 17 0
umem_np 786432B 786432B 786432B 21 0
ksyms 2182106B 2445312B 2445312B 296 0
ctf 2145463B 2158592B 2158592B 338 0
heap_core 1765376B 931115008B 0B 57 0
heaptext 10715136B 67108864B 0B 255 0
module_text 10464984B 10723328B 10715136B 331 0
id32 0B 0B 0B 0 0
module_data 913855B 2072576B 1765376B 448 0
logminor_space 25B 262137B 0B 113 0
taskq_id_arena 83B 2147483647B 0B 162 0
zfs_file_data 3156045824B 3181379584B 0B 154761 0
zfs_file_data_buf 2504310784B 2504310784B 2504310784B 8577092 0
device 1536000B 1073741824B 0B 32959 0
segkp 27553792B 2147483648B 0B 4444 0
mac_minor_ids 3B 130070B 0B 4 0
rctl_ids 39B 32767B 0B 39 0
zoneid_space 0B 9998B 0B 0 0
taskid_space 41B 999999B 0B 147 0
pool_ids 0B 999998B 0B 0 0
contracts 44B 2147483646B 0B 153 0
ip_minor_arena_sa 128B 262140B 0B 2 0
ip_minor_arena_la 64B 4294705152B 0B 1 0
lib_va_32 7483392B 2138820608B 0B 20 0
lport-instances 2B 65536B 0B 2 0
rport-instances 7B 65536B 0B 7 0
tl_minor_space 40B 262138B 0B 137 0
keysock 0B 4294967295B 0B 0 0
spdsock 64B 4294967295B 0B 1 0
namefs_inodes 128B 65536B 0B 2 0
lib_va_64 15134720B 137988695961600B 0B 68 0
Hex0xffffff024e6a4558_minor 0B 4294967294B 0B 0 0
Hex0xffffff024e6a4560_minor 0B 4294967294B 0B 0 0
devfsadm_event_channel 1B 101B 0B 1 0
devfsadm_event_channel 1B 2B 0B 1 0
syseventconfd_door 0B 101B 0B 0 0
syseventconfd_door 1B 2B 0B 1 0
syseventd_channel 3B 101B 0B 3 0
syseventd_channel 1B 2B 0B 1 0
dtrace 68449B 4294967295B 0B 77712 0
dtrace_minor 0B 4294967293B 0B 3 0
logdmux_minor 0B 256B 0B 0 0
ptms_minor 2B 16B 0B 12 0
sppptun_minor 0B 16B 0B 0 0
lmsysid_space 1B 16383B 0B 1 0
crypto 0B 16B 0B 26 0
------------------------- ---------- ----------- ---------- --------- -----
Updated by George Wilson about 11 years ago
- Category set to zfs - Zettabyte File System
- Assignee set to George Wilson
As Bryan indicated this is a case of memory fragmentation. The proposed fix makes the arc evict memory when the free space in the arena becomes scarce and provides a new VM interface to reap quantum caches.
Updated by George Wilson about 11 years ago
- Subject changed from zfs causing system to hang in vmem_xalloc() to zfs causing system to hang in vmem_xalloc()
Updated by Piotr Jasiukajtis about 11 years ago
- Status changed from New to Closed
- % Done changed from 0 to 100
Fixed in changeset 13682:e7836650181b
1618 zfs causing system to hang in vmem_xalloc()
Reviewed by: Eric Schrock <eric.schrock@delphix.com>
Reviewed by: Adam Leventhal <ahl@delphix.com>
Reviewed by: Matt Ahrens <matt@delphix.com>
Reviewed by: Richard Elling <richard.elling@richardelling.com>
Reviewed by: Garrett D'Amore <garrett@damore.org>
Reviewed by: Steve Gonczi <gonczi@comcast.net>
Approved by: Gordon Ross <gwr@nexenta.com>
Actions