Bug #7366
vioif happily creates rx descriptors until it consumes all memory
Start date:
2016-09-09
Due date:
% Done:
100%
Estimated time:
Difficulty:
Hard
Tags:
Description
We debugged a case today where a VM with 4 GB of memory was hanging. First, let's look at ::stacks from this machine:
THREAD STATE SOBJ COUNT
ffffff0007a11c40 SLEEP CV 580
swtch+0x141
cv_wait+0x70
taskq_thread_wait+0xbe
taskq_thread+0x37c
thread_start+8
ffffff025185bb40 SLEEP CV 55
swtch+0x141
cv_wait_sig_swap_core+0x1b9
cv_wait_sig_swap+0x17
cv_waituntil_sig+0xbd
lwp_park+0x15e
syslwp_park+0x63
_sys_sysenter_post_swapgs+0x153
ffffff024e9467c0 SLEEP CV 29
swtch+0x141
cv_wait+0x70
page_create_throttle+0x16a
page_create_va+0x5d4
pvn_read_kluster+0x10c
ufs_getpage_miss+0x107
ufs_getpage+0x8a6
fop_getpage+0x7e
segvn_fault+0xdfa
as_fault+0x312
pagefault+0x96
trap+0x30c
ffffff024eabeae0 SLEEP SHUTTLE 28
swtch+0x141
shuttle_swtch+0x203
door_return+0x214
doorfs32+0x180
_sys_sysenter_post_swapgs+0x153
ffffff0007a9ec40 FREE <NONE> 27
<no consistent stack found>
ffffff024e5dc480 SLEEP CV 20
swtch+0x141
cv_timedwait_sig_hires+0x39d
cv_waituntil_sig+0xfa
nanosleep+0x19f
_sys_sysenter_post_swapgs+0x153
ffffff0007cd5c40 SLEEP CV 19
swtch+0x141
cv_wait+0x70
evch_delivery_hold+0x70
evch_delivery_thr+0x2c4
thread_start+8
ffffff02515347e0 SLEEP CV 17
swtch+0x141
cv_wait_sig_swap_core+0x1b9
cv_wait_sig_swap+0x17
cv_timedwait_sig_hrtime+0x35
poll_common+0x288
pollsys+0x2dc
_sys_sysenter_post_swapgs+0x153
ffffff024ea3f0e0 SLEEP SHUTTLE 15
swtch_to+0xb6
shuttle_resume+0x2af
door_return+0x3e0
doorfs32+0x180
_sys_sysenter_post_swapgs+0x153
ffffff0250f443e0 SLEEP MUTEX 14
swtch+0x141
turnstile_block+0x21a
mutex_vector_enter+0x3a3
txg_hold_open+0x55
dmu_tx_try_assign+0x87
dmu_tx_assign+0x60
zfs_write+0x4a6
fop_write+0xf3
do_io+0x9e
rfs4_op_write+0x433
rfs4_compound+0x2ed
rfs4_dispatch+0x264
common_dispatch+0xb2f
rfs_dispatch+0x2d
svc_getreq+0x1c1
svc_run+0xe0
svc_do_run+0x8e
nfssys+0x111
_sys_sysenter_post_swapgs+0x153
ffffff024e54b8a0 SLEEP SHUTTLE 13
swtch_to+0xb6
shuttle_resume+0x2af
door_return+0x3e0
doorfs32+0x180
sys_syscall32+0x123
ffffff024ea3f820 SLEEP CV 12
swtch+0x141
cv_wait_sig_swap_core+0x1b9
cv_wait_sig_swap+0x17
pause+0x45
_sys_sysenter_post_swapgs+0x153
ffffff0007dc5c40 SLEEP CV 9
swtch+0x141
cv_wait+0x70
squeue_polling_thread+0xa9
thread_start+8
ffffff0008b62c40 SLEEP CV 9
swtch+0x141
cv_wait+0x70
squeue_worker+0x114
thread_start+8
ffffff00089c8c40 SLEEP CV 8
swtch+0x141
cv_timedwait_hires+0xec
cv_timedwait_sig_hires+0x1eb
cv_reltimedwait_sig+0x4f
reaper_thread+0xbc
thread_start+8
ffffff02516993a0 SLEEP CV 8
<thread not in memory>
ffffff0007ad4c40 FREE <NONE> 6
av_dispatch_softvect+0x60
apix_dispatch_softint+0x41
ffffff000b2c5c40 SLEEP CV 6
swtch+0x141
cv_timedwait_hires+0xec
cv_reltimedwait+0x51
taskq_thread_wait+0x64
taskq_d_thread+0x145
thread_start+8
ffffff024e511440 SLEEP CV 6
swtch+0x141
cv_wait_sig_swap_core+0x1b9
cv_wait_sig_swap+0x17
cv_waituntil_sig+0xbd
lwp_park+0x15e
syslwp_park+0x63
sys_syscall32+0x123
ffffff025162d0a0 SLEEP CV 5
swtch+0x141
cv_timedwait_hires+0xec
cv_timedwait+0x5c
overlay_target_lookup_request+0x7e
overlay_target_ioctl+0x102
cdev_ioctl+0x39
spec_ioctl+0x60
fop_ioctl+0x55
ioctl+0x9b
_sys_sysenter_post_swapgs+0x153
ffffff0251af4820 SLEEP CV 5
swtch+0x141
cv_wait_sig_swap_core+0x1b9
cv_wait_sig_swap+0x17
sigsuspend+0x101
_sys_sysenter_post_swapgs+0x153
ffffff000c56cc40 RUN <NONE> 5
swtch+0x141
cv_wait+0x70
taskq_thread_wait+0xbe
taskq_thread+0x37c
thread_start+8
ffffff0008d35c40 SLEEP CV 4
swtch+0x141
cv_timedwait_hires+0xec
cv_reltimedwait+0x51
kcfpool_svc+0x84
thread_start+8
ffffff025185b400 SLEEP CV 4
swtch+0x141
cv_wait_sig_swap_core+0x1b9
cv_wait_sig_swap+0x17
cv_waituntil_sig+0xbd
sigtimedwait+0x19c
_sys_sysenter_post_swapgs+0x153
ffffff024e5a8440 SLEEP SHUTTLE 4
swtch+0x141
shuttle_swtch+0x203
door_return+0x214
doorfs32+0x180
sys_syscall32+0x123
ffffff0008ba4c40 SLEEP CV 3
swtch+0x141
cv_wait+0x70
mac_soft_ring_worker+0xb1
thread_start+8
ffffff02570930a0 SLEEP CV 3
swtch+0x141
cv_wait+0x70
page_create_throttle+0x16a
page_create_va+0x5d4
swap_getapage+0x113
pvn_getpages+0x161
swap_getpage+0x61
fop_getpage+0x7e
anon_getpage+0x18d
segvn_faultpage+0x168
segvn_fault+0x8e6
as_fault+0x312
pagefault+0x96
trap+0x30c
ffffff0007ce1c40 ONPROC <NONE> 3
swtch+0x141
cpu_pause+0x80
thread_start+8
ffffff024e9a1b80 SLEEP CV 2
swtch+0x141
cv_timedwait_hires+0xec
txg_delay+0xd3
dsl_dir_tempreserve_space+0x150
dmu_tx_try_assign+0x212
dmu_tx_assign+0x60
zfs_write+0x4a6
fop_write+0xf3
do_io+0x9e
rfs4_op_write+0x433
rfs4_compound+0x2ed
rfs4_dispatch+0x264
common_dispatch+0xb2f
rfs_dispatch+0x2d
svc_getreq+0x1c1
svc_run+0xe0
svc_do_run+0x8e
nfssys+0x111
_sys_sysenter_post_swapgs+0x153
ffffff0256109b20 SLEEP CV 2
swtch+0x141
cv_timedwait_sig_hires+0x39d
cv_timedwait_sig_hrtime+0x2a
poll_common+0xea
pollsys+0x2dc
_sys_sysenter_post_swapgs+0x153
ffffff024ea8fc40 SLEEP CV 2
swtch+0x141
cv_timedwait_sig_hires+0x39d
cv_timedwait_sig_hrtime+0x2a
poll_common+0x288
pollsys+0x2dc
_sys_sysenter_post_swapgs+0x153
ffffff024e5a87e0 SLEEP CV 2
swtch+0x141
cv_timedwait_sig_hires+0x39d
cv_waituntil_sig+0xfa
lwp_park+0x15e
syslwp_park+0x63
_sys_sysenter_post_swapgs+0x153
ffffff0008209c40 SLEEP CV 2
swtch+0x141
cv_wait+0x70
mac_srs_worker+0x141
thread_start+8
ffffff0007c0fc40 SLEEP CV 2
swtch+0x141
cv_wait+0x70
progressbar_key_abort_thread+0x73
thread_start+8
ffffff000c372c40 SLEEP CV 2
swtch+0x141
cv_wait+0x70
spa_thread+0x1db
thread_start+8
ffffff000b2e7c40 SLEEP CV 2
swtch+0x141
cv_wait+0x70
svc_thread_creator+0xd8
thread_start+8
ffffff0007e2fc40 SLEEP CV 2
swtch+0x141
cv_wait+0x70
txg_thread_wait+0xaf
txg_quiesce_thread+0x106
thread_start+8
ffffff024e481780 SLEEP CV 2
swtch+0x141
cv_wait_sig+0x185
cte_get_event+0xb3
ctfs_endpoint_ioctl+0xf9
ctfs_bu_ioctl+0x4b
fop_ioctl+0x55
ioctl+0x9b
sys_syscall32+0x123
ffffff024e5fac40 SLEEP CV 2
swtch+0x141
cv_wait_sig+0x185
door_unref+0x94
doorfs32+0xf5
_sys_sysenter_post_swapgs+0x153
ffffff024e50d0c0 SLEEP CV 2
swtch+0x141
cv_wait_sig+0x185
svc_wait+0xb7
nfssys+0xd1
_sys_sysenter_post_swapgs+0x153
ffffff0256f1cc40 SLEEP SHUTTLE 2
swtch_to+0xb6
shuttle_resume+0x2af
door_call+0x336
doorfs32+0xa7
_sys_sysenter_post_swapgs+0x153
ffffff0007cedc40 ONPROC <NONE> 2
apix_intr_exit+0x24
apix_hilevel_intr_epilog+0xc1
apix_do_interrupt+0x34a
_sys_rtt_ints_disabled+8
default_lock_delay+0x50
apix_do_interrupt+0xfe
_interrupt+0xba
mach_cpu_idle+6
cpu_idle+0x11a
idle+0xa7
thread_start+8
ffffff0007d3ec40 FREE <NONE> 1
apic_send_ipi+0x73
send_dirint+0x18
cbe_xcall+0xac
cyclic_reprogram_here+0x46
cyclic_reprogram+0x68
lbolt_ev_to_cyclic+0xcf
av_dispatch_softvect+0x60
apix_dispatch_softint+0x41
ffffff0007e43c40 FREE <NONE> 1
checked_rdmsr+0x42
ntv_rdmsr_xc+0x8c
call_func_ntv+0x130
ntv_rdmsr+0x3f
0xffffff024d2e3c48
gcpu_mca_logout+0x28a
gcpu_ntv_mca_poll+0x1fc
0xffffff024ca6fd40
cyclic_reprogram_here+0x46
apix_post_hardint+0x34
0xffffff024d1d6000
apix_dispatch_pending_hardint+0x3f
ffffff00083abc40 SLEEP CV 1
swtch+0x141
cv_timedwait_hires+0xec
arc_reclaim_thread+0x153
thread_start+8
ffffff000b8b0c40 SLEEP CV 1
swtch+0x141
cv_timedwait_hires+0xec
cv_reltimedwait+0x51
idm_wd_thread+0x203
thread_start+8
ffffff0007b8ec40 SLEEP CV 1
swtch+0x141
cv_timedwait_hires+0xec
cv_reltimedwait+0x51
kcfpoold+0xf6
thread_start+8
ffffff0007b4cc40 SLEEP CV 1
swtch+0x141
cv_timedwait_hires+0xec
cv_reltimedwait+0x51
page_capture_thread+0xb1
thread_start+8
ffffff0007f37c40 SLEEP CV 1
swtch+0x141
cv_timedwait_hires+0xec
cv_reltimedwait+0x51
seg_pasync_thread+0xd1
thread_start+8
ffffff00083a5c40 SLEEP CV 1
swtch+0x141
cv_timedwait_hires+0xec
cv_reltimedwait+0x51
stmf_svc_timeout+0x112
stmf_svc+0x1c0
taskq_thread+0x2d0
thread_start+8
ffffff0007bd9c40 SLEEP CV 1
swtch+0x141
cv_timedwait_hires+0xec
cv_timedwait+0x5c
dce_reclaim_worker+0xab
thread_start+8
ffffff00083c8c40 SLEEP CV 1
swtch+0x141
cv_timedwait_hires+0xec
cv_timedwait+0x5c
l2arc_feed_thread+0xad
thread_start+8
ffffff000899ec40 SLEEP CV 1
swtch+0x141
cv_timedwait_hires+0xec
cv_timedwait+0x5c
nlm_gc+0x5c
thread_start+8
ffffff000b3eac40 SLEEP CV 1
swtch+0x141
cv_timedwait_hires+0xec
cv_timedwait+0x5c
txg_thread_wait+0x5f
txg_sync_thread+0x111
thread_start+8
ffffff00083b7c40 SLEEP CV 1
swtch+0x141
cv_timedwait_hires+0xec
dbuf_evict_thread+0xef
thread_start+8
ffffff024d2d4b00 SLEEP CV 1
swtch+0x141
cv_timedwait_sig_hires+0x39d
cv_timedwait_sig_hrtime+0x2a
poll_common+0x288
pollsys+0x2dc
sys_syscall32+0x123
ffffff024ea38100 SLEEP CV 1
swtch+0x141
cv_timedwait_sig_hires+0x39d
cv_waituntil_sig+0xfa
lwp_park+0x15e
syslwp_park+0x63
sys_syscall32+0x123
ffffff0255b62020 SLEEP CV 1
swtch+0x141
cv_timedwait_sig_hires+0x39d
cv_waituntil_sig+0xfa
nanosleep+0x19f
ffffff0007fecc40 SLEEP CV 1
swtch+0x141
cv_wait+0x70
au_output_thread+0x80
taskq_thread+0x2d0
thread_start+8
ffffff000b3f0c40 SLEEP CV 1
swtch+0x141
cv_wait+0x70
crypto_bufcall_service+0x8d
thread_start+8
fffffffffbc31d80 SLEEP CV 1
swtch+0x141
cv_wait+0x70
dbuf_read+0x1ff
dmu_tx_check_ioerr+0x76
dmu_tx_count_write+0x49c
dmu_tx_hold_write+0x5a
zvol_strategy+0x1e5
bdev_strategy+0x61
spec_startio+0x8e
spec_pageio+0x3a
fop_pageio+0x59
swap_putapage+0x176
swap_putpage+0x295
fop_putpage+0x4c
segkp_unlock+0x21b
segkp_fault+0xfe
swapout+0x269
sched+0x8f7
main+0x46c
ffffff0007b88c40 SLEEP CV 1
swtch+0x141
cv_wait+0x70
dld_taskq_dispatch+0x115
thread_start+8
ffffff0007f2bc40 SLEEP CV 1
swtch+0x141
cv_wait+0x70
fsflush+0x21d
thread_start+8
ffffff000b8a4c40 SLEEP CV 1
swtch+0x141
cv_wait+0x70
i_mac_notify_thread+0xee
thread_start+8
ffffff0007bdfc40 SLEEP CV 1
swtch+0x141
cv_wait+0x70
ill_taskq_dispatch+0x173
thread_start+8
ffffff0007bc7c40 SLEEP CV 1
swtch+0x141
cv_wait+0x70
ipsec_loader+0x149
thread_start+8
ffffff0008f40c40 SLEEP CV 1
swtch+0x141
cv_wait+0x70
irm_balance_thread+0x50
thread_start+8
ffffff0008d7dc40 SLEEP CV 1
swtch+0x141
cv_wait+0x70
log_event_deliver+0x1b3
thread_start+8
ffffff0007f31c40 SLEEP CV 1
swtch+0x141
cv_wait+0x70
mod_uninstall_daemon+0x123
thread_start+8
ffffff0008984c40 SLEEP CV 1
swtch+0x141
cv_wait+0x70
nfsauth_refresh_thread+0x205
thread_start+8
ffffff0251534b80 SLEEP CV 1
swtch+0x141
cv_wait+0x70
page_create_throttle+0x16a
page_create_va+0x5d4
swap_getapage+0x113
pvn_getpages+0x161
swap_getpage+0x61
fop_getpage+0x7e
anon_zero+0xb6
segvn_lockop+0xd1a
as_ctl+0x2c5
as_map_locked+0x22b
as_map+0x58
brk_internal+0x2b0
brk+0x51
_sys_sysenter_post_swapgs+0x153
ffffff0007ae6c40 SLEEP CV 1
swtch+0x141
cv_wait+0x70
page_create_wait+0x137
page_create_io+0x85
page_create_io_wrapper+0x47
segkmem_xalloc+0x8b
segkmem_alloc_io_64G+0x32
vmem_xalloc+0x5b1
vmem_alloc+0x135
kalloca+0x181
i_ddi_mem_alloc+0x15d
ddi_dma_mem_alloc+0x205
vioif_rx_construct+0xb0
kmem_cache_alloc+0x26b
vioif_add_rx+0x10d
vioif_populate_rx+0x1c
vioif_rx_handler+0x2d
apix_dispatch_pending_autovect+0x101
apix_dispatch_pending_hardint+0x34
ffffff0007b5ec40 SLEEP CV 1
swtch+0x141
cv_wait+0x70
pm_dep_thread+0xbd
thread_start+8
ffffff0007a23c40 SLEEP CV 1
swtch+0x141
cv_wait+0x70
scsi_hba_barrier_daemon+0xd6
thread_start+8
ffffff0007a29c40 SLEEP CV 1
swtch+0x141
cv_wait+0x70
scsi_lunchg1_daemon+0x1f4
thread_start+8
ffffff0007a2fc40 SLEEP CV 1
swtch+0x141
cv_wait+0x70
scsi_lunchg2_daemon+0x121
thread_start+8
ffffff000b2d5c40 SLEEP CV 1
swtch+0x141
cv_wait+0x70
softmac_taskq_dispatch+0x11d
thread_start+8
ffffff0007b3ac40 SLEEP CV 1
swtch+0x141
cv_wait+0x70
streams_bufcall_service+0x8d
thread_start+8
ffffff0007b40c40 SLEEP CV 1
swtch+0x141
cv_wait+0x70
streams_qbkgrnd_service+0x151
thread_start+8
ffffff0007b46c40 SLEEP CV 1
swtch+0x141
cv_wait+0x70
streams_sqbkgrnd_service+0xe5
thread_start+8
ffffff0007ccfc40 SLEEP CV 1
swtch+0x141
cv_wait+0x70
task_commit+0xd9
thread_start+8
ffffff0007a0bc40 SLEEP CV 1
swtch+0x141
cv_wait+0x70
thread_reaper+0xb9
thread_start+8
ffffff0007b52c40 SLEEP CV 1
swtch+0x141
cv_wait+0x70
ufs_thread_idle+0x147
thread_start+8
ffffff0007b58c40 SLEEP CV 1
swtch+0x141
cv_wait+0x70
ufs_thread_run+0x80
ufs_thread_hlock+0x73
thread_start+8
ffffff0255b72120 SLEEP CV 1
swtch+0x141
cv_wait+0x70
zio_wait+0x6b
dmu_buf_hold_array_by_dnode+0x169
dmu_read_uio_dnode+0x5a
dmu_read_uio_dbuf+0x51
zfs_read+0x1a3
fop_read+0xf3
pread+0x1a3
pread32+0x24
_sys_sysenter_post_swapgs+0x153
ffffff0007f1fc40 SLEEP CV 1
swtch+0x141
cv_wait+0x70
zio_wait+0x6b
dmu_tx_count_write+0x645
dmu_tx_hold_write+0x5a
zvol_strategy+0x1e5
bdev_strategy+0x61
spec_startio+0x8e
spec_pageio+0x3a
fop_pageio+0x59
swap_putapage+0x176
swap_putpage+0x295
fop_putpage+0x4c
pageout+0x26a
thread_start+8
ffffff0007df5c40 SLEEP CV 1
swtch+0x141
cv_wait+0x70
zio_wait+0x6b
dsl_pool_sync_mos+0x4a
dsl_pool_sync+0x25e
spa_sync+0x386
txg_sync_thread+0x227
thread_start+8
ffffff024e4e6b40 SLEEP CV 1
swtch+0x141
cv_wait_sig+0x185
door_unref+0x94
doorfs32+0xf5
sys_syscall32+0x123
ffffff0251c124e0 SLEEP CV 1
swtch+0x141
cv_wait_sig+0x185
so_dequeue_msg+0x317
so_recvmsg+0x296
socket_recvmsg+0x33
socket_vop_read+0x5f
fop_read+0xf3
read+0x2a7
read32+0x1e
_sys_sysenter_post_swapgs+0x153
ffffff02516a1480 SLEEP CV 1
swtch+0x141
cv_wait_sig+0x185
str_cv_wait+0x27
strwaitq+0x2c3
strgetmsg+0x22b
msgio32+0x23a
getmsg32+0x98
_sys_sysenter_post_swapgs+0x153
ffffff024e518420 SLEEP CV 1
swtch+0x141
cv_wait_sig_swap_core+0x1b9
cv_wait_sig_swap+0x17
cv_waituntil_sig+0xbd
port_getn+0x3cf
portfs+0x25d
portfs32+0x78
sys_syscall32+0x123
ffffff024e481040 SLEEP CV 1
swtch+0x141
cv_wait_sig_swap_core+0x1b9
cv_wait_sig_swap+0x17
cv_waituntil_sig+0xbd
sigtimedwait+0x19c
sys_syscall32+0x123
ffffff024e481b20 SLEEP CV 1
swtch+0x141
cv_wait_sig_swap_core+0x1b9
cv_wait_sig_swap+0x17
sigsuspend+0x101
sys_syscall32+0x123
ffffff0250d96b00 SLEEP CV 1
swtch+0x141
cv_wait_sig_swap_core+0x1b9
cv_wait_sig_swap+0x17
waitid+0x315
waitsys32+0x36
_sys_sysenter_post_swapgs+0x153
ffffff024e511b80 SLEEP SHUTTLE 1
swtch_to+0xb6
shuttle_resume+0x2af
door_call+0x336
doorfs32+0xa7
sys_syscall32+0x123
ffffff000a913c40 RUN <NONE> 1
swtch+0x141
cv_timedwait_hires+0xec
cv_reltimedwait+0x51
taskq_thread_wait+0x64
taskq_d_thread+0x145
thread_start+8
ffffff0007f25c40 RUN <NONE> 1
swtch+0x141
cv_wait+0x70
pageout_scanner+0x121
thread_start+8
ffffff0007e5cc40 ONPROC <NONE> 1
apix_intr_exit+0x24
apix_hilevel_intr_epilog+0xc1
apix_do_interrupt+0x34a
_sys_rtt_ints_disabled+8
mutex_delay_default+7
apix_do_interrupt+0xfe
_interrupt+0xba
mach_cpu_idle+6
cpu_idle+0x11a
idle+0xa7
thread_start+8
ffffff0007d5cc40 STOPPED <NONE> 1
<no consistent stack found>
ffffff0007a05c40 PANIC <NONE> 1
1
There's a lot of stuff in page_create_throttle() and friends. If we look at ::memstat and ::kmastat we can see the bleak picture painted for us.
> ::memstat
Page Summary Pages MB %Tot
------------ ---------------- ---------------- ----
Kernel 861908 3366 82%
Boot pages 68306 266 7%
ZFS File Data 92003 359 9%
Anon 22249 86 2%
Exec and libs 193 0 0%
Page cache 1471 5 0%
Free (cachelist) 0 0 0%
Free (freelist) 3 0 0%
Total 1046133 4086
Physical 1046132 4086
> ::kmastat
cache buf buf buf memory alloc alloc
name size in use total in use succeed fail
------------------------------ ----- --------- --------- ------ ---------- -----
kmem_magazine_1 16 509 6275 100K 35328 0
kmem_magazine_3 32 368 3125 100K 6650 0
kmem_magazine_7 64 290 2046 132K 11255 0
kmem_magazine_15 128 233 2728 352K 19811 0
kmem_magazine_31 256 64 465 124K 1381 0
kmem_magazine_47 384 0 20 8K 89 0
kmem_magazine_63 512 0 21 12K 101 0
kmem_magazine_95 768 8 30 24K 203 0
kmem_magazine_143 1.12K 0 0 0 0 0
kmem_slab_cache 72 25453 35640 2.53M 54323 0
kmem_bufctl_cache 24 97454 97695 2.29M 132298 0
kmem_bufctl_audit_cache 192 0 0 0 0 0
kmem_va_4096 4K 16738 16768 65.5M 30623 0
kmem_va_8192 8K 15703 15712 123M 15805 0
kmem_va_12288 12K 322 330 4.12M 326 0
kmem_va_16384 16K 274 464 7.25M 1458 0
kmem_va_20480 20K 21 24 512K 23 0
kmem_va_24576 24K 24 25 640K 25 0
kmem_va_28672 28K 14 16 512K 15 0
kmem_va_32768 32K 74 76 2.38M 81 0
kmem_alloc_8 8 41165 51306 408K 828042 0
kmem_alloc_16 16 11051 15562 248K 202270 0
kmem_alloc_24 24 53920 60120 1.41M 990041 0
kmem_alloc_32 32 25369 28250 904K 933843 0
kmem_alloc_40 40 51362 51500 2.01M 253719 0
kmem_alloc_48 48 3385 9047 436K 606957 0
kmem_alloc_56 56 3673 3834 216K 37055 0
kmem_alloc_64 64 34161 35154 2.21M 842001 0
kmem_alloc_80 80 5583 10100 808K 156509 0
kmem_alloc_96 96 762 861 84K 70310 0
kmem_alloc_112 112 62100 62125 6.93M 350970 0
kmem_alloc_128 128 444 1116 144K 291126 0
kmem_alloc_160 160 1440 1700 272K 29369 0
kmem_alloc_192 192 452 660 132K 105399 0
kmem_alloc_224 224 736 799 188K 32597 0
kmem_alloc_256 256 15661 16470 4.29M 463054 0
kmem_alloc_320 320 45139 45168 14.7M 126745 0
kmem_alloc_384 384 976 1220 488K 17612 0
kmem_alloc_448 448 129 144 72K 4178 0
kmem_alloc_512 512 272 288 144K 10649 0
kmem_alloc_640 640 3028 3072 2M 34977 0
kmem_alloc_768 768 59 80 64K 2143 0
kmem_alloc_896 896 362 378 336K 5488 0
kmem_alloc_1152 1.12K 676 679 776K 66696 0
kmem_alloc_1344 1.31K 47 51 68K 568 0
kmem_alloc_1600 1.56K 42 50 80K 1743 0
kmem_alloc_2048 2K 494 546 1.07M 12084 0
kmem_alloc_2688 2.62K 131 150 400K 3077 0
kmem_alloc_4096 4K 574 579 2.26M 32223 0
kmem_alloc_8192 8K 146 153 1.20M 5437 0
kmem_alloc_12288 12K 281 287 3.36M 5428 0
kmem_alloc_16384 16K 32 38 608K 6299 0
kmem_alloc_24576 24K 16 19 456K 78 0
kmem_alloc_32768 32K 64 68 2.12M 1751 0
kmem_alloc_40960 40K 2 6 240K 20 0
kmem_alloc_49152 48K 0 5 240K 17 0
kmem_alloc_57344 56K 1 7 392K 22 0
kmem_alloc_65536 64K 8 11 704K 20 0
kmem_alloc_73728 72K 2 4 288K 8 0
kmem_alloc_81920 80K 0 3 240K 9 0
kmem_alloc_90112 88K 0 3 264K 4 0
kmem_alloc_98304 96K 2 4 384K 8 0
kmem_alloc_106496 104K 0 4 416K 9 0
kmem_alloc_114688 112K 0 4 448K 5 0
kmem_alloc_122880 120K 2 4 480K 5 0
kmem_alloc_131072 128K 8 15 1.88M 48834 0
streams_mblk 64 44414 44516 2.80M 130888 0
streams_dblk_16 128 59 93 12K 5805 0
streams_dblk_80 192 241 560 112K 18797 0
streams_dblk_144 256 2 45 12K 115701 0
streams_dblk_208 320 204 432 144K 4229 0
streams_dblk_272 384 0 50 20K 3976 0
streams_dblk_336 448 1 24 12K 149 0
streams_dblk_528 640 0 96 64K 38550 0
streams_dblk_1040 1.12K 0 35 40K 200 0
streams_dblk_1488 1.56K 0 10 16K 33 0
streams_dblk_1936 2K 0 18 36K 4611 0
streams_dblk_2576 2.62K 1 12 32K 121 0
streams_dblk_3856 3.88K 0 3 12K 9 0
streams_dblk_8192 112 0 31 4K 31 0
streams_dblk_12048 11.9K 0 3 36K 18 0
streams_dblk_16384 112 0 31 4K 2 0
streams_dblk_20240 19.9K 0 0 0 0 0
streams_dblk_24576 112 0 0 0 0 0
streams_dblk_28432 27.9K 0 0 0 0 0
streams_dblk_32768 112 0 0 0 0 0
streams_dblk_36624 35.9K 0 0 0 0 0
streams_dblk_40960 112 0 31 4K 1 0
streams_dblk_44816 43.9K 0 1 44K 1 0
streams_dblk_49152 112 0 0 0 0 0
streams_dblk_53008 51.9K 0 1 52K 1 0
streams_dblk_57344 112 0 31 4K 1 0
streams_dblk_61200 59.9K 0 0 0 0 0
streams_dblk_65536 112 0 0 0 0 0
streams_dblk_69392 67.9K 0 1 68K 1 0
streams_dblk_73728 112 0 31 4K 1 0
streams_dblk_esb 112 43532 43617 5.50M 328187 146
streams_fthdr 408 0 0 0 0 0
streams_ftblk 376 0 0 0 0 0
multidata 248 0 0 0 0 0
multidata_pdslab 6.95K 0 0 0 0 0
multidata_pattbl 32 0 0 0 0 0
log_cons_cache 48 11 83 4K 21 0
taskq_ent_cache 56 1302 1562 88K 40374 0
taskq_cache 280 135 154 44K 480 0
kmem_io_64G_128 128 0 0 0 0 0
kmem_io_64G_256 256 0 0 0 0 0
kmem_io_64G_512 512 0 0 0 0 0
kmem_io_64G_1024 1K 0 0 0 0 0
kmem_io_64G_2048 2K 0 0 0 0 0
kmem_io_64G_4096 4K 1024 1024 4M 1024 0
kmem_io_4G_128 128 0 0 0 0 0
kmem_io_4G_256 256 1 15 4K 18 0
kmem_io_4G_512 512 1 8 4K 2 0
kmem_io_4G_1024 1K 0 0 0 0 0
kmem_io_4G_2048 2K 0 0 0 0 0
kmem_io_4G_4096 4K 0 0 0 0 0
kmem_io_2G_128 128 0 0 0 0 0
kmem_io_2G_256 256 0 0 0 0 0
kmem_io_2G_512 512 0 0 0 0 0
kmem_io_2G_1024 1K 0 0 0 0 0
kmem_io_2G_2048 2K 0 0 0 0 0
kmem_io_2G_4096 4K 0 0 0 0 0
kmem_io_16M_128 128 0 0 0 0 0
kmem_io_16M_256 256 0 0 0 0 0
kmem_io_16M_512 512 0 0 0 0 0
kmem_io_16M_1024 1K 0 0 0 0 0
kmem_io_16M_2048 2K 0 0 0 0 0
kmem_io_16M_4096 4K 0 0 0 0 0
id32_cache 32 0 0 0 0 0
bp_map_4096 4K 0 0 0 0 0
bp_map_8192 8K 0 16 128K 11901 0
bp_map_12288 12K 0 10 128K 2 0
bp_map_16384 16K 0 0 0 0 0
bp_map_20480 20K 0 6 128K 2 0
bp_map_24576 24K 0 0 0 0 0
bp_map_28672 28K 0 0 0 0 0
bp_map_32768 32K 0 0 0 0 0
htable_t 72 2996 3025 220K 40624 0
hment_t 64 14727 28210 1.78M 1906098 0
hat_t 176 56 110 20K 5212 0
HatHash 8K 2 7 56K 767 0
HatVlpHash 4K 54 57 228K 5716 0
zfs_file_data_4096 4K 13 32 128K 42 0
zfs_file_data_8192 8K 719 720 5.62M 2795 0
zfs_file_data_12288 12K 10 20 256K 26 0
zfs_file_data_16384 16K 2 8 128K 4 0
zfs_file_data_20480 20K 14 30 640K 35 0
zfs_file_data_24576 24K 5 10 256K 6 0
zfs_file_data_28672 28K 10 16 512K 15 0
zfs_file_data_32768 32K 3 20 640K 85 0
segkp_4096 4K 53 64 256K 3335 0
segkp_8192 8K 0 0 0 0 0
segkp_12288 12K 0 0 0 0 0
segkp_16384 16K 0 0 0 0 0
segkp_20480 20K 0 0 0 0 0
umem_np_4096 4K 0 32 128K 165 0
umem_np_8192 8K 0 16 128K 68 0
umem_np_12288 12K 0 0 0 0 0
umem_np_16384 16K 0 8 128K 83 0
umem_np_20480 20K 0 6 128K 6 0
umem_np_24576 24K 0 5 128K 4 0
umem_np_28672 28K 0 0 0 0 0
umem_np_32768 32K 0 4 128K 68 0
mod_hash_entries 24 279 501 12K 11079 0
ipp_mod 304 0 0 0 0 0
ipp_action 368 0 0 0 0 0
ipp_packet 64 0 0 0 0 0
seg_cache 96 3070 3198 312K 214631 0
seg_pcache 104 0 38 4K 49 0
fnode_cache 176 3 20 4K 23 0
pipe_cache 320 13 36 12K 2056 0
snode_cache 152 252 286 44K 10744 0
dv_node_cache 176 50 242 44K 510 0
mac_impl_cache 13.4K 1 8 108K 1 0
mac_ring_cache 192 0 0 0 0 0
flow_entry_cache 26.7K 5 7 188K 7 0
flow_tab_cache 216 1 18 4K 1 0
mac_soft_ring_cache 376 3 10 4K 9 0
mac_srs_cache 3.16K 2 5 16K 6 0
mac_bcast_grp_cache 80 4 50 4K 6 0
mac_client_impl_cache 1.36K 1 8 12K 1 0
mac_promisc_impl_cache 120 0 0 0 0 0
dls_link_cache 352 1 11 4K 1 0
dls_devnet_cache 368 1 10 4K 1 0
sdev_node_cache 272 551 560 160K 1317 0
sdev_plugin 144 9 27 4K 9 0
dev_info_node_cache 680 141 156 104K 1198 0
ndi_fm_entry_cache 32 0 0 0 0 0
thread_cache 920 274 568 568K 5329 0
lwp_cache 1.73K 733 756 1.31M 1161 0
turnstile_cache 64 1018 2170 140K 9798 0
tslabel_cache 48 2 83 4K 2 0
cred_cache 200 137 180 36K 12136 0
rctl_cache 48 1007 1162 56K 47972 0
rctl_val_cache 64 1909 2108 136K 103117 0
task_cache 160 43 50 8K 94 0
kmem_defrag_cache 208 2 19 4K 2 0
kmem_move_cache 56 0 0 0 0 0
rootnex_dmahdl 2.53K 45743 45753 119M 54478 0
ddi_periodic 168 0 0 0 0 0
cyclic_id_cache 72 24 55 4K 32 0
callout_cachebabecafe 80 402 403 52K 402 0
callout_lcachebabecafe 48 727 744 48K 727 0
dnlc_space_cache 24 0 0 0 0 0
vfs_cache 208 47 57 12K 48 0
vn_cache 216 6398 6675 1.74M 16896 0
vsk_anchor_cache 40 20 100 4K 23 0
file_cache 64 423 558 36K 57519 0
stream_head_cache 376 128 150 60K 4832 0
queue_cache 656 310 330 220K 5786 0
syncq_cache 160 7 25 4K 25 0
qband_cache 64 2 62 4K 2 0
linkinfo_cache 48 4 83 4K 5 0
ciputctrl_cache 256 0 0 0 0 0
serializer_cache 64 23 62 4K 90 0
as_cache 224 55 102 24K 5211 0
marker_cache 120 0 33 4K 8252 0
anon_cache 48 23902 24485 1.15M 439375 0
anonmap_cache 112 2196 2275 260K 134741 0
segvn_cache 168 3015 3128 544K 195224 0
segvn_szc_cache1 4K 0 0 0 0 0
flk_edges 48 0 0 0 0 0
fdb_cache 104 0 0 0 0 0
timer_cache 144 2 27 4K 2 0
vmu_bound_cache 56 0 0 0 0 0
vmu_object_cache 64 0 0 0 0 0
physio_buf_cache 248 0 16 4K 90 0
process_cache 3.92K 60 66 264K 3745 0
ufs_inode_cache 368 2590 2590 1.01M 2590 0
directio_buf_cache 272 0 0 0 0 0
lufs_save 24 0 0 0 0 0
lufs_bufs 256 0 0 0 0 0
lufs_mapentry_cache 112 0 0 0 0 0
bootfs_node_cache 256 1 15 4K 1 0
audit_proc 40 61 100 4K 3353 0
drv_secobj_cache 296 0 0 0 0 0
dld_str_cache 304 3 13 4K 11 0
kcf_sreq_cache 56 0 0 0 0 0
kcf_areq_cache 296 0 0 0 0 0
kcf_context_cache 112 0 0 0 0 0
ip_minor_arena_sa_1 1 32 64 64 190 0
ip_minor_arena_la_1 1 7 64 64 123 0
ip_conn_cache 720 0 5 4K 2 0
tcp_conn_cache 1.77K 26 30 60K 65 0
udp_conn_cache 1.23K 32 39 52K 370 0
rawip_conn_cache 1.09K 0 0 0 0 0
rts_conn_cache 816 4 9 8K 9 0
ire_cache 352 18 33 12K 59 0
ncec_cache 200 3 20 4K 39 0
nce_cache 88 3 45 4K 47 0
rt_entry 152 10 26 4K 51 0
radix_mask 32 3 125 4K 5 0
radix_node 120 2 33 4K 2 0
ipsec_actions 72 0 0 0 0 0
ipsec_selectors 80 0 0 0 0 0
ipsec_policy 80 0 0 0 0 0
tcp_timercache 88 37 90 8K 245 0
tcp_notsack_blk_cache 24 0 0 0 0 0
squeue_cache 168 9 20 4K 9 0
sctp_conn_cache 2.47K 0 0 0 0 0
sctp_faddr_cache 176 0 0 0 0 0
sctp_set_cache 24 0 0 0 0 0
sctp_ftsn_set_cache 16 0 0 0 0 0
dce_cache 152 4 26 4K 4 0
ire_gw_secattr_cache 24 0 0 0 0 0
socket_cache 656 30 42 28K 250 0
socktpi_cache 960 0 4 4K 1 0
socktpi_unix_cache 960 15 32 32K 77 0
sock_sod_cache 648 0 0 0 0 0
exacct_object_cache 40 0 0 0 0 0
kssl_cache 1.59K 0 0 0 0 0
clnt_clts_endpnt_cache 88 0 0 0 0 0
range_seg_cache 64 1346 1488 96K 5765 0
zio_cache 936 28 344 344K 287859 0
zio_link_cache 48 16 2656 128K 143617 0
zio_buf_512 512 850 864 432K 29100 0
zio_data_buf_512 512 0 24 12K 386 0
zio_buf_1024 1K 222 368 368K 7984 0
zio_data_buf_1024 1K 0 8 8K 109 0
zio_buf_1536 1.50K 11 40 60K 255 0
zio_data_buf_1536 1.50K 0 8 12K 99 0
zio_buf_2048 2K 32 48 96K 1038 0
zio_data_buf_2048 2K 0 8 16K 106 0
zio_buf_2560 2.50K 22 40 100K 298 0
zio_data_buf_2560 2.50K 0 24 60K 108 0
zio_buf_3072 3K 17 32 96K 174 0
zio_data_buf_3072 3K 0 16 48K 81 0
zio_buf_3584 3.50K 25 32 112K 113 0
zio_data_buf_3584 3.50K 0 16 56K 76 0
zio_buf_4096 4K 15 20 80K 1730 0
zio_data_buf_4096 4K 0 4 16K 67 0
zio_buf_5120 5K 8 16 80K 300 0
zio_data_buf_5120 5K 0 8 40K 126 0
zio_buf_6144 6K 17 24 144K 150 0
zio_data_buf_6144 6K 0 4 24K 54 0
zio_buf_7168 7K 1 8 56K 90 0
zio_data_buf_7168 7K 0 8 56K 40 0
zio_buf_8192 8K 1 6 48K 165 0
zio_data_buf_8192 8K 719 719 5.62M 3178 0
zio_buf_10240 10K 7 14 140K 142 0
zio_data_buf_10240 10K 0 8 80K 43 0
zio_buf_12288 12K 1 6 72K 116 0
zio_data_buf_12288 12K 0 3 36K 23 0
zio_buf_14336 14K 1 6 84K 75 0
zio_data_buf_14336 14K 0 4 56K 19 0
zio_buf_16384 16K 140 151 2.36M 5221 0
zio_data_buf_16384 16K 0 2 32K 20 0
zio_buf_20480 20K 1 5 100K 60 0
zio_data_buf_20480 20K 1 5 100K 30 0
zio_buf_24576 24K 0 5 120K 27 0
zio_data_buf_24576 24K 1 5 120K 29 0
zio_buf_28672 28K 3 5 140K 19 0
zio_data_buf_28672 28K 0 4 112K 31 0
zio_buf_32768 32K 1 6 192K 17 0
zio_data_buf_32768 32K 0 3 96K 107 0
zio_buf_40960 40K 2 8 320K 51 0
zio_data_buf_40960 40K 1 7 280K 395 0
zio_buf_49152 48K 18 18 864K 39 0
zio_data_buf_49152 48K 3 6 288K 286 0
zio_buf_57344 56K 0 3 168K 4 0
zio_data_buf_57344 56K 29 31 1.70M 253 0
zio_buf_65536 64K 0 2 128K 7 0
zio_data_buf_65536 64K 609 611 38.2M 3598 0
zio_buf_81920 80K 0 6 480K 26 0
zio_data_buf_81920 80K 82 85 6.64M 112 0
zio_buf_98304 96K 0 7 672K 52 0
zio_data_buf_98304 96K 0 3 288K 9 0
zio_buf_114688 112K 0 6 672K 243 0
zio_data_buf_114688 112K 0 3 336K 4 0
zio_buf_131072 128K 5 13 1.62M 9607 0
zio_data_buf_131072 128K 2434 2441 305M 51825 0
zio_buf_163840 160K 0 0 0 0 0
zio_data_buf_163840 160K 0 0 0 0 0
zio_buf_196608 192K 0 0 0 0 0
zio_data_buf_196608 192K 0 0 0 0 0
zio_buf_229376 224K 0 0 0 0 0
zio_data_buf_229376 224K 0 0 0 0 0
zio_buf_262144 256K 0 0 0 0 0
zio_data_buf_262144 256K 0 0 0 0 0
zio_buf_327680 320K 0 0 0 0 0
zio_data_buf_327680 320K 0 0 0 0 0
zio_buf_393216 384K 0 0 0 0 0
zio_data_buf_393216 384K 0 0 0 0 0
zio_buf_458752 448K 0 0 0 0 0
zio_data_buf_458752 448K 0 0 0 0 0
zio_buf_524288 512K 0 0 0 0 0
zio_data_buf_524288 512K 0 0 0 0 0
zio_buf_655360 640K 0 0 0 0 0
zio_data_buf_655360 640K 0 0 0 0 0
zio_buf_786432 768K 0 0 0 0 0
zio_data_buf_786432 768K 0 0 0 0 0
zio_buf_917504 896K 0 0 0 0 0
zio_data_buf_917504 896K 0 0 0 0 0
zio_buf_1048576 1M 0 0 0 0 0
zio_data_buf_1048576 1M 0 0 0 0 0
zio_buf_1310720 1.25M 0 0 0 0 0
zio_data_buf_1310720 1.25M 0 0 0 0 0
zio_buf_1572864 1.50M 0 0 0 0 0
zio_data_buf_1572864 1.50M 0 0 0 0 0
zio_buf_1835008 1.75M 0 0 0 0 0
zio_data_buf_1835008 1.75M 0 0 0 0 0
zio_buf_2097152 2M 0 0 0 0 0
zio_data_buf_2097152 2M 0 0 0 0 0
zio_buf_2621440 2.50M 0 0 0 0 0
zio_data_buf_2621440 2.50M 0 0 0 0 0
zio_buf_3145728 3M 0 0 0 0 0
zio_data_buf_3145728 3M 0 0 0 0 0
zio_buf_3670016 3.50M 0 0 0 0 0
zio_data_buf_3670016 3.50M 0 0 0 0 0
zio_buf_4194304 4M 0 0 0 0 0
zio_data_buf_4194304 4M 0 0 0 0 0
zio_buf_5242880 5M 0 0 0 0 0
zio_data_buf_5242880 5M 0 0 0 0 0
zio_buf_6291456 6M 0 0 0 0 0
zio_data_buf_6291456 6M 0 0 0 0 0
zio_buf_7340032 7M 0 0 0 0 0
zio_data_buf_7340032 7M 0 0 0 0 0
zio_buf_8388608 8M 0 0 0 0 0
zio_data_buf_8388608 8M 0 0 0 0 0
zio_buf_10485760 10M 0 0 0 0 0
zio_data_buf_10485760 10M 0 0 0 0 0
zio_buf_12582912 12M 0 0 0 0 0
zio_data_buf_12582912 12M 0 0 0 0 0
zio_buf_14680064 14M 0 0 0 0 0
zio_data_buf_14680064 14M 0 0 0 0 0
zio_buf_16777216 16M 0 0 0 0 0
zio_data_buf_16777216 16M 0 0 0 0 0
sa_cache 120 398 429 52K 777 0
dnode_t 624 950 962 592K 2806 0
arc_buf_hdr_t_full 176 4860 9394 1.67M 45564 0
arc_buf_hdr_t_l2only 88 0 0 0 0 0
arc_buf_t 32 1536 2375 76K 45635 0
dmu_buf_impl_t 200 2155 2980 596K 47949 0
zil_lwb_cache 192 2 40 8K 69 0
zfs_znode_cache 248 398 400 100K 777 0
tl_cache 432 33 54 24K 285 0
keysock_1 1 0 0 0 0 0
spdsock_1 1 0 64 64 1 0
namefs_inodes_1 1 21 64 64 21 0
port_cache 80 18 50 4K 72 0
lnode_cache 32 3 125 4K 11 0
softmac_cache 568 1 7 4K 1 0
softmac_upper_cache 232 0 0 0 0 0
dtrace_state_cache 16K 0 0 0 0 0
blkdev0_xfer 152 0 52 8K 4338 0
blkdev1_xfer 152 0 52 8K 28963 0
vioif0_rx 80 43747 43900 3.43M 304081 146
idm_buf_cache 240 0 0 0 0 0
idm_task_cache 2.38K 0 0 0 0 0
idm_tx_pdu_cache 464 0 0 0 0 0
idm_rx_pdu_cache 513 0 0 0 0 0
idm_128k_buf_cache 128K 0 0 0 0 0
gsqueue_cb_cache 24 0 0 0 0 0
gsqueue_cpu_cache 24 4 167 4K 4 0
squeue_set_cache 64 1 62 4K 1 0
frameio_cache 784 0 0 0 0 0
vnd_str_cache 1.41K 0 0 0 0 0
vnd_dev_cache 192 0 0 0 0 0
vnd_pnsd_cache 264 1 15 4K 1 0
pty_map 64 1 62 4K 1 0
sppptun_map 440 0 0 0 0 0
authkern_cache 72 3 55 4K 4 0
authnone_cache 72 0 0 0 0 0
authloopback_cache 72 0 0 0 0 0
authdes_cache_handle 80 0 0 0 0 0
rnode_cache 656 0 0 0 0 0
nfs_access_cache 56 0 0 0 0 0
client_handle_cache 32 0 0 0 0 0
rnode4_cache 968 0 0 0 0 0
svnode_cache 40 0 0 0 0 0
nfs4_access_cache 56 0 0 0 0 0
client_handle4_cache 32 0 0 0 0 0
nfs4_ace4vals_cache 48 0 0 0 0 0
nfs4_ace4_list_cache 264 0 0 0 0 0
NFS_idmap_cache 48 4 83 4K 4 0
nlm_host_cache 240 0 0 0 0 0
nlm_vhold_cache 48 0 0 0 0 0
nlm_rpch_cache 32 0 0 0 0 0
nfslog_small_rec 512 0 0 0 0 0
nfslog_medium_rec 8K 0 0 0 0 0
nfslog_large_rec 32K 0 0 0 0 0
exi_cache_handle 120 3 33 4K 3 0
nfs_xuio_cache 664 0 0 0 0 0
Hex0xffffff024cf2f488_minor_1 1 0 0 0 0 0
Hex0xffffff024cf2f490_minor_1 1 0 0 0 0 0
overlay_plugin_cache 72 0 0 0 0 0
overlay_target 88 0 0 0 0 0
overlay_entry 184 0 0 0 0 0
Client_entry_cache 736 1 11 8K 1 0
ClntIP_entry_cache 376 0 0 0 0 0
OpenOwner_entry_cache 560 1 7 4K 1 0
OpenStateID_entry_cache 264 32 45 12K 32 0
LockStateID_entry_cache 528 0 0 0 0 0
Lockowner_entry_cache 176 0 0 0 0 0
File_entry_cache 296 31 39 12K 31 0
DelegStateID_entry_cache 216 0 36 8K 22 0
audit_buffer 152 0 26 4K 94 0
------------------------------ ----- --------- --------- ------ ---------- -----
Total [hat_memload] 1.99M 1946722 0
Total [kmem_msb] 5.65M 261441 0
Total [kmem_va] 204M 48356 0
Total [kmem_default] 211M 9615106 292
Total [kmem_io_64G] 4M 1024 0
Total [kmem_io_4G] 8K 20 0
Total [bp_map] 384K 11905 0
Total [umem_np] 768K 394 0
Total [zfs_file_data] 8.12M 3008 0
Total [zfs_file_data_buf] 359M 61214 0
Total [segkp] 256K 3335 0
Total [ip_minor_arena_sa] 64 190 0
Total [ip_minor_arena_la] 64 123 0
Total [spdsock] 64 1 0
Total [namefs_inodes] 64 21 0
------------------------------ ----- --------- --------- ------ ---------- -----
vmem memory memory memory alloc alloc
name in use total import succeed fail
------------------------------ --------- ---------- --------- ---------- -----
heap 3.15G 1014G 0 56891 0
vmem_metadata 19.0M 19.1M 19.1M 4652 0
vmem_seg 17.8M 17.8M 17.8M 4539 0
vmem_hash 1000K 1004K 1004K 49 0
vmem_vmem 288K 320K 284K 92 0
static 0 0 0 0 0
static_alloc 0 0 0 0 0
hat_memload 1.99M 1.99M 1.99M 564 0
kstat 617K 652K 588K 2324 0
kmem_metadata 6.88M 8.12M 8.12M 2109 0
kmem_msb 5.65M 5.65M 5.65M 1912 0
kmem_cache 369K 472K 472K 544 0
kmem_hash 764K 780K 780K 341 4
kmem_log 193K 196K 196K 9 0
kmem_firewall_va 48.6M 48.6M 48.6M 138 0
kmem_firewall 0 0 0 0 0
kmem_oversize 48.6M 48.6M 48.6M 138 0
mod_sysfile 1.43K 4K 4K 39 0
kmem_va 215M 215M 215M 3492 0
kmem_default 211M 211M 211M 42479 292
kmem_io_64G 2.69G 2.85G 2.85G 51050 0
kmem_io_4G 8K 8K 8K 414 0
kmem_io_2G 0 0 0 0 0
kmem_io_16M 0 0 0 0 0
bp_map 496K 496K 496K 363 0
umem_np 768K 768K 768K 78 0
ksyms 1.88M 2.09M 2.09M 257 0
ctf 1.17M 1.44M 1.44M 249 0
heap_core 1.39M 888M 0 46 0
heaptext 7.92M 64M 0 182 0
module_text 7.98M 8.15M 7.92M 256 0
id32 0 0 0 0 0
module_data 1.03M 1.69M 1.39M 354 0
logminor_space 27 256K 0 37 0
taskq_id_arena 99 2.00G 0 386 0
zfs_file_data 361M 5.99G 0 24647 0
zfs_file_data_buf 359M 359M 359M 26424 0
device 1.20M 1G 0 44 0
segkp 24.6M 2G 0 5525 0
mac_minor_ids 5 127K 0 6 0
rctl_ids 44 32.0K 0 44 0
zoneid_space 0 9.76K 0 0 0
taskid_space 43 977K 0 94 0
pool_ids 0 977K 0 0 0
contracts 47 2.00G 0 106 0
ddi_periodic 0 1023 0 0 0
bootfs_minors 1 2.00G 0 1 0
ip_minor_arena_sa 64 256K 0 1 0
ip_minor_arena_la 64 4.00G 0 1 0
lib_va_32 7.73M 1.99G 0 20 0
lport-instances 0 64K 0 0 0
rport-instances 0 64K 0 0 0
tl_minor_space 33 256K 0 256 0
keysock 0 4.00G 0 0 0
spdsock 64 4.00G 0 1 0
namefs_inodes 64 64K 0 1 0
lofi_id 1 64.0M 0 1 0
lib_va_64 14.0M 125T 0 53 0
dtrace 72 4.00G 0 15229 0
dtrace_minor 0 4.00G 0 0 0
vnd_minors 0 2.00G 0 0 0
logdmux_minor 0 256 0 0 0
ptms_minor 1 16 0 1 0
ipf_minor 1 4.00G 0 23 0
sppptun_minor 0 16 0 0 0
devfsadm_event_channel 1 101 0 1 0
devfsadm_event_channel 1 2 0 1 0
syseventconfd_door 0 101 0 0 0
syseventconfd_door 1 2 0 1 0
syseventd_channel 1 101 0 1 0
syseventd_channel 1 2 0 1 0
Hex0xffffff024cf2f488_minor 0 4.00G 0 0 0
Hex0xffffff024cf2f490_minor 0 4.00G 0 0 0
overlay_target_minors 1 2.00G 0 1 0
Client_id_space 1 128K 0 1 0
ClntIP_id_space 0 1M 0 0 0
OpenOwner_id_space 1 1M 0 1 0
OpenStateID_id_space 32 1M 0 32 0
LockStateID_id_space 0 1M 0 0 0
Lockowner_id_space 0 1M 0 0 0
DelegStateID_id_space 0 1M 0 22 0
------------------------------ --------- ---------- --------- ---------- -----
Okay, so the kmem_io_64G has a large amount of memory in it. This seems high for a 4 GiB VM, so where is that memory. When I looked at this, I then turned my attention to the vioif driver, since it has an interrupt thread stack trying to allocate memory. All of the vioif descriptors come out of a single kmem_cache: vioif0_rx. There are 43900 entries here. If we look at the cache, it's allocating DMA memory for each one equal to its max payload, give or take a few bytes. So in this case 64k. If we do the math, 43900 entries times about 64k is about 2.8 GiB! In many cases our MTU is much less than that and only about 1500 bytes or 9k; however, keeping the maximum size here does simplify the virtio implementation.
Okay, so with this knowledge, the question is why does this happen. Well, if we look at the virtio code it initially will fill up its entire descriptor pool, nothing wrong with that. It's not actually that many worth in the queue originally. Usually this number is on the order of 32-256 entries. We're a lot more than that. What happens is that the virtio net driver has no notion of backpressure for rx descriptors. It basically will happily allocate memory until the system runs out of it and constantly loan them up. Instead, we should have a fixed amount of descriptors we'll loan up and then move on to reloading. Otherwise we basically always risk running out of kernel memory by receiving more data than we can process.
Related issues
History
Updated by
Updated by Joshua M. Clulow 8 months ago
- Related to Feature #11329: improved Virtio framework added
Updated by Electric Monk 8 months ago
Updated by - Status changed from New to Closed
- % Done changed from 0 to 100
git commit f8296c60994fb27105f37ac6f75661e4a6bdbab7
commit f8296c60994fb27105f37ac6f75661e4a6bdbab7
Author: Joshua M. Clulow <josh@sysmgr.org>
Date: 2019-08-16T17:37:47.000Z
11329 improved Virtio framework
10012 vioblk should not accept an all-zero serial number
7366 vioif happily creates rx descriptors until it consumes all memory
Reviewed by: Robert Mustacchi <rm@joyent.com>
Approved by: Richard Lowe <richlowe@richlowe.net>