Bug #7366
closedvioif happily creates rx descriptors until it consumes all memory
100%
Description
We debugged a case today where a VM with 4 GB of memory was hanging. First, let's look at ::stacks
from this machine:
THREAD STATE SOBJ COUNT ffffff0007a11c40 SLEEP CV 580 swtch+0x141 cv_wait+0x70 taskq_thread_wait+0xbe taskq_thread+0x37c thread_start+8 ffffff025185bb40 SLEEP CV 55 swtch+0x141 cv_wait_sig_swap_core+0x1b9 cv_wait_sig_swap+0x17 cv_waituntil_sig+0xbd lwp_park+0x15e syslwp_park+0x63 _sys_sysenter_post_swapgs+0x153 ffffff024e9467c0 SLEEP CV 29 swtch+0x141 cv_wait+0x70 page_create_throttle+0x16a page_create_va+0x5d4 pvn_read_kluster+0x10c ufs_getpage_miss+0x107 ufs_getpage+0x8a6 fop_getpage+0x7e segvn_fault+0xdfa as_fault+0x312 pagefault+0x96 trap+0x30c ffffff024eabeae0 SLEEP SHUTTLE 28 swtch+0x141 shuttle_swtch+0x203 door_return+0x214 doorfs32+0x180 _sys_sysenter_post_swapgs+0x153 ffffff0007a9ec40 FREE <NONE> 27 <no consistent stack found> ffffff024e5dc480 SLEEP CV 20 swtch+0x141 cv_timedwait_sig_hires+0x39d cv_waituntil_sig+0xfa nanosleep+0x19f _sys_sysenter_post_swapgs+0x153 ffffff0007cd5c40 SLEEP CV 19 swtch+0x141 cv_wait+0x70 evch_delivery_hold+0x70 evch_delivery_thr+0x2c4 thread_start+8 ffffff02515347e0 SLEEP CV 17 swtch+0x141 cv_wait_sig_swap_core+0x1b9 cv_wait_sig_swap+0x17 cv_timedwait_sig_hrtime+0x35 poll_common+0x288 pollsys+0x2dc _sys_sysenter_post_swapgs+0x153 ffffff024ea3f0e0 SLEEP SHUTTLE 15 swtch_to+0xb6 shuttle_resume+0x2af door_return+0x3e0 doorfs32+0x180 _sys_sysenter_post_swapgs+0x153 ffffff0250f443e0 SLEEP MUTEX 14 swtch+0x141 turnstile_block+0x21a mutex_vector_enter+0x3a3 txg_hold_open+0x55 dmu_tx_try_assign+0x87 dmu_tx_assign+0x60 zfs_write+0x4a6 fop_write+0xf3 do_io+0x9e rfs4_op_write+0x433 rfs4_compound+0x2ed rfs4_dispatch+0x264 common_dispatch+0xb2f rfs_dispatch+0x2d svc_getreq+0x1c1 svc_run+0xe0 svc_do_run+0x8e nfssys+0x111 _sys_sysenter_post_swapgs+0x153 ffffff024e54b8a0 SLEEP SHUTTLE 13 swtch_to+0xb6 shuttle_resume+0x2af door_return+0x3e0 doorfs32+0x180 sys_syscall32+0x123 ffffff024ea3f820 SLEEP CV 12 swtch+0x141 cv_wait_sig_swap_core+0x1b9 cv_wait_sig_swap+0x17 pause+0x45 _sys_sysenter_post_swapgs+0x153 ffffff0007dc5c40 SLEEP CV 9 swtch+0x141 cv_wait+0x70 squeue_polling_thread+0xa9 thread_start+8 ffffff0008b62c40 SLEEP CV 9 swtch+0x141 cv_wait+0x70 squeue_worker+0x114 thread_start+8 ffffff00089c8c40 SLEEP CV 8 swtch+0x141 cv_timedwait_hires+0xec cv_timedwait_sig_hires+0x1eb cv_reltimedwait_sig+0x4f reaper_thread+0xbc thread_start+8 ffffff02516993a0 SLEEP CV 8 <thread not in memory> ffffff0007ad4c40 FREE <NONE> 6 av_dispatch_softvect+0x60 apix_dispatch_softint+0x41 ffffff000b2c5c40 SLEEP CV 6 swtch+0x141 cv_timedwait_hires+0xec cv_reltimedwait+0x51 taskq_thread_wait+0x64 taskq_d_thread+0x145 thread_start+8 ffffff024e511440 SLEEP CV 6 swtch+0x141 cv_wait_sig_swap_core+0x1b9 cv_wait_sig_swap+0x17 cv_waituntil_sig+0xbd lwp_park+0x15e syslwp_park+0x63 sys_syscall32+0x123 ffffff025162d0a0 SLEEP CV 5 swtch+0x141 cv_timedwait_hires+0xec cv_timedwait+0x5c overlay_target_lookup_request+0x7e overlay_target_ioctl+0x102 cdev_ioctl+0x39 spec_ioctl+0x60 fop_ioctl+0x55 ioctl+0x9b _sys_sysenter_post_swapgs+0x153 ffffff0251af4820 SLEEP CV 5 swtch+0x141 cv_wait_sig_swap_core+0x1b9 cv_wait_sig_swap+0x17 sigsuspend+0x101 _sys_sysenter_post_swapgs+0x153 ffffff000c56cc40 RUN <NONE> 5 swtch+0x141 cv_wait+0x70 taskq_thread_wait+0xbe taskq_thread+0x37c thread_start+8 ffffff0008d35c40 SLEEP CV 4 swtch+0x141 cv_timedwait_hires+0xec cv_reltimedwait+0x51 kcfpool_svc+0x84 thread_start+8 ffffff025185b400 SLEEP CV 4 swtch+0x141 cv_wait_sig_swap_core+0x1b9 cv_wait_sig_swap+0x17 cv_waituntil_sig+0xbd sigtimedwait+0x19c _sys_sysenter_post_swapgs+0x153 ffffff024e5a8440 SLEEP SHUTTLE 4 swtch+0x141 shuttle_swtch+0x203 door_return+0x214 doorfs32+0x180 sys_syscall32+0x123 ffffff0008ba4c40 SLEEP CV 3 swtch+0x141 cv_wait+0x70 mac_soft_ring_worker+0xb1 thread_start+8 ffffff02570930a0 SLEEP CV 3 swtch+0x141 cv_wait+0x70 page_create_throttle+0x16a page_create_va+0x5d4 swap_getapage+0x113 pvn_getpages+0x161 swap_getpage+0x61 fop_getpage+0x7e anon_getpage+0x18d segvn_faultpage+0x168 segvn_fault+0x8e6 as_fault+0x312 pagefault+0x96 trap+0x30c ffffff0007ce1c40 ONPROC <NONE> 3 swtch+0x141 cpu_pause+0x80 thread_start+8 ffffff024e9a1b80 SLEEP CV 2 swtch+0x141 cv_timedwait_hires+0xec txg_delay+0xd3 dsl_dir_tempreserve_space+0x150 dmu_tx_try_assign+0x212 dmu_tx_assign+0x60 zfs_write+0x4a6 fop_write+0xf3 do_io+0x9e rfs4_op_write+0x433 rfs4_compound+0x2ed rfs4_dispatch+0x264 common_dispatch+0xb2f rfs_dispatch+0x2d svc_getreq+0x1c1 svc_run+0xe0 svc_do_run+0x8e nfssys+0x111 _sys_sysenter_post_swapgs+0x153 ffffff0256109b20 SLEEP CV 2 swtch+0x141 cv_timedwait_sig_hires+0x39d cv_timedwait_sig_hrtime+0x2a poll_common+0xea pollsys+0x2dc _sys_sysenter_post_swapgs+0x153 ffffff024ea8fc40 SLEEP CV 2 swtch+0x141 cv_timedwait_sig_hires+0x39d cv_timedwait_sig_hrtime+0x2a poll_common+0x288 pollsys+0x2dc _sys_sysenter_post_swapgs+0x153 ffffff024e5a87e0 SLEEP CV 2 swtch+0x141 cv_timedwait_sig_hires+0x39d cv_waituntil_sig+0xfa lwp_park+0x15e syslwp_park+0x63 _sys_sysenter_post_swapgs+0x153 ffffff0008209c40 SLEEP CV 2 swtch+0x141 cv_wait+0x70 mac_srs_worker+0x141 thread_start+8 ffffff0007c0fc40 SLEEP CV 2 swtch+0x141 cv_wait+0x70 progressbar_key_abort_thread+0x73 thread_start+8 ffffff000c372c40 SLEEP CV 2 swtch+0x141 cv_wait+0x70 spa_thread+0x1db thread_start+8 ffffff000b2e7c40 SLEEP CV 2 swtch+0x141 cv_wait+0x70 svc_thread_creator+0xd8 thread_start+8 ffffff0007e2fc40 SLEEP CV 2 swtch+0x141 cv_wait+0x70 txg_thread_wait+0xaf txg_quiesce_thread+0x106 thread_start+8 ffffff024e481780 SLEEP CV 2 swtch+0x141 cv_wait_sig+0x185 cte_get_event+0xb3 ctfs_endpoint_ioctl+0xf9 ctfs_bu_ioctl+0x4b fop_ioctl+0x55 ioctl+0x9b sys_syscall32+0x123 ffffff024e5fac40 SLEEP CV 2 swtch+0x141 cv_wait_sig+0x185 door_unref+0x94 doorfs32+0xf5 _sys_sysenter_post_swapgs+0x153 ffffff024e50d0c0 SLEEP CV 2 swtch+0x141 cv_wait_sig+0x185 svc_wait+0xb7 nfssys+0xd1 _sys_sysenter_post_swapgs+0x153 ffffff0256f1cc40 SLEEP SHUTTLE 2 swtch_to+0xb6 shuttle_resume+0x2af door_call+0x336 doorfs32+0xa7 _sys_sysenter_post_swapgs+0x153 ffffff0007cedc40 ONPROC <NONE> 2 apix_intr_exit+0x24 apix_hilevel_intr_epilog+0xc1 apix_do_interrupt+0x34a _sys_rtt_ints_disabled+8 default_lock_delay+0x50 apix_do_interrupt+0xfe _interrupt+0xba mach_cpu_idle+6 cpu_idle+0x11a idle+0xa7 thread_start+8 ffffff0007d3ec40 FREE <NONE> 1 apic_send_ipi+0x73 send_dirint+0x18 cbe_xcall+0xac cyclic_reprogram_here+0x46 cyclic_reprogram+0x68 lbolt_ev_to_cyclic+0xcf av_dispatch_softvect+0x60 apix_dispatch_softint+0x41 ffffff0007e43c40 FREE <NONE> 1 checked_rdmsr+0x42 ntv_rdmsr_xc+0x8c call_func_ntv+0x130 ntv_rdmsr+0x3f 0xffffff024d2e3c48 gcpu_mca_logout+0x28a gcpu_ntv_mca_poll+0x1fc 0xffffff024ca6fd40 cyclic_reprogram_here+0x46 apix_post_hardint+0x34 0xffffff024d1d6000 apix_dispatch_pending_hardint+0x3f ffffff00083abc40 SLEEP CV 1 swtch+0x141 cv_timedwait_hires+0xec arc_reclaim_thread+0x153 thread_start+8 ffffff000b8b0c40 SLEEP CV 1 swtch+0x141 cv_timedwait_hires+0xec cv_reltimedwait+0x51 idm_wd_thread+0x203 thread_start+8 ffffff0007b8ec40 SLEEP CV 1 swtch+0x141 cv_timedwait_hires+0xec cv_reltimedwait+0x51 kcfpoold+0xf6 thread_start+8 ffffff0007b4cc40 SLEEP CV 1 swtch+0x141 cv_timedwait_hires+0xec cv_reltimedwait+0x51 page_capture_thread+0xb1 thread_start+8 ffffff0007f37c40 SLEEP CV 1 swtch+0x141 cv_timedwait_hires+0xec cv_reltimedwait+0x51 seg_pasync_thread+0xd1 thread_start+8 ffffff00083a5c40 SLEEP CV 1 swtch+0x141 cv_timedwait_hires+0xec cv_reltimedwait+0x51 stmf_svc_timeout+0x112 stmf_svc+0x1c0 taskq_thread+0x2d0 thread_start+8 ffffff0007bd9c40 SLEEP CV 1 swtch+0x141 cv_timedwait_hires+0xec cv_timedwait+0x5c dce_reclaim_worker+0xab thread_start+8 ffffff00083c8c40 SLEEP CV 1 swtch+0x141 cv_timedwait_hires+0xec cv_timedwait+0x5c l2arc_feed_thread+0xad thread_start+8 ffffff000899ec40 SLEEP CV 1 swtch+0x141 cv_timedwait_hires+0xec cv_timedwait+0x5c nlm_gc+0x5c thread_start+8 ffffff000b3eac40 SLEEP CV 1 swtch+0x141 cv_timedwait_hires+0xec cv_timedwait+0x5c txg_thread_wait+0x5f txg_sync_thread+0x111 thread_start+8 ffffff00083b7c40 SLEEP CV 1 swtch+0x141 cv_timedwait_hires+0xec dbuf_evict_thread+0xef thread_start+8 ffffff024d2d4b00 SLEEP CV 1 swtch+0x141 cv_timedwait_sig_hires+0x39d cv_timedwait_sig_hrtime+0x2a poll_common+0x288 pollsys+0x2dc sys_syscall32+0x123 ffffff024ea38100 SLEEP CV 1 swtch+0x141 cv_timedwait_sig_hires+0x39d cv_waituntil_sig+0xfa lwp_park+0x15e syslwp_park+0x63 sys_syscall32+0x123 ffffff0255b62020 SLEEP CV 1 swtch+0x141 cv_timedwait_sig_hires+0x39d cv_waituntil_sig+0xfa nanosleep+0x19f ffffff0007fecc40 SLEEP CV 1 swtch+0x141 cv_wait+0x70 au_output_thread+0x80 taskq_thread+0x2d0 thread_start+8 ffffff000b3f0c40 SLEEP CV 1 swtch+0x141 cv_wait+0x70 crypto_bufcall_service+0x8d thread_start+8 fffffffffbc31d80 SLEEP CV 1 swtch+0x141 cv_wait+0x70 dbuf_read+0x1ff dmu_tx_check_ioerr+0x76 dmu_tx_count_write+0x49c dmu_tx_hold_write+0x5a zvol_strategy+0x1e5 bdev_strategy+0x61 spec_startio+0x8e spec_pageio+0x3a fop_pageio+0x59 swap_putapage+0x176 swap_putpage+0x295 fop_putpage+0x4c segkp_unlock+0x21b segkp_fault+0xfe swapout+0x269 sched+0x8f7 main+0x46c ffffff0007b88c40 SLEEP CV 1 swtch+0x141 cv_wait+0x70 dld_taskq_dispatch+0x115 thread_start+8 ffffff0007f2bc40 SLEEP CV 1 swtch+0x141 cv_wait+0x70 fsflush+0x21d thread_start+8 ffffff000b8a4c40 SLEEP CV 1 swtch+0x141 cv_wait+0x70 i_mac_notify_thread+0xee thread_start+8 ffffff0007bdfc40 SLEEP CV 1 swtch+0x141 cv_wait+0x70 ill_taskq_dispatch+0x173 thread_start+8 ffffff0007bc7c40 SLEEP CV 1 swtch+0x141 cv_wait+0x70 ipsec_loader+0x149 thread_start+8 ffffff0008f40c40 SLEEP CV 1 swtch+0x141 cv_wait+0x70 irm_balance_thread+0x50 thread_start+8 ffffff0008d7dc40 SLEEP CV 1 swtch+0x141 cv_wait+0x70 log_event_deliver+0x1b3 thread_start+8 ffffff0007f31c40 SLEEP CV 1 swtch+0x141 cv_wait+0x70 mod_uninstall_daemon+0x123 thread_start+8 ffffff0008984c40 SLEEP CV 1 swtch+0x141 cv_wait+0x70 nfsauth_refresh_thread+0x205 thread_start+8 ffffff0251534b80 SLEEP CV 1 swtch+0x141 cv_wait+0x70 page_create_throttle+0x16a page_create_va+0x5d4 swap_getapage+0x113 pvn_getpages+0x161 swap_getpage+0x61 fop_getpage+0x7e anon_zero+0xb6 segvn_lockop+0xd1a as_ctl+0x2c5 as_map_locked+0x22b as_map+0x58 brk_internal+0x2b0 brk+0x51 _sys_sysenter_post_swapgs+0x153 ffffff0007ae6c40 SLEEP CV 1 swtch+0x141 cv_wait+0x70 page_create_wait+0x137 page_create_io+0x85 page_create_io_wrapper+0x47 segkmem_xalloc+0x8b segkmem_alloc_io_64G+0x32 vmem_xalloc+0x5b1 vmem_alloc+0x135 kalloca+0x181 i_ddi_mem_alloc+0x15d ddi_dma_mem_alloc+0x205 vioif_rx_construct+0xb0 kmem_cache_alloc+0x26b vioif_add_rx+0x10d vioif_populate_rx+0x1c vioif_rx_handler+0x2d apix_dispatch_pending_autovect+0x101 apix_dispatch_pending_hardint+0x34 ffffff0007b5ec40 SLEEP CV 1 swtch+0x141 cv_wait+0x70 pm_dep_thread+0xbd thread_start+8 ffffff0007a23c40 SLEEP CV 1 swtch+0x141 cv_wait+0x70 scsi_hba_barrier_daemon+0xd6 thread_start+8 ffffff0007a29c40 SLEEP CV 1 swtch+0x141 cv_wait+0x70 scsi_lunchg1_daemon+0x1f4 thread_start+8 ffffff0007a2fc40 SLEEP CV 1 swtch+0x141 cv_wait+0x70 scsi_lunchg2_daemon+0x121 thread_start+8 ffffff000b2d5c40 SLEEP CV 1 swtch+0x141 cv_wait+0x70 softmac_taskq_dispatch+0x11d thread_start+8 ffffff0007b3ac40 SLEEP CV 1 swtch+0x141 cv_wait+0x70 streams_bufcall_service+0x8d thread_start+8 ffffff0007b40c40 SLEEP CV 1 swtch+0x141 cv_wait+0x70 streams_qbkgrnd_service+0x151 thread_start+8 ffffff0007b46c40 SLEEP CV 1 swtch+0x141 cv_wait+0x70 streams_sqbkgrnd_service+0xe5 thread_start+8 ffffff0007ccfc40 SLEEP CV 1 swtch+0x141 cv_wait+0x70 task_commit+0xd9 thread_start+8 ffffff0007a0bc40 SLEEP CV 1 swtch+0x141 cv_wait+0x70 thread_reaper+0xb9 thread_start+8 ffffff0007b52c40 SLEEP CV 1 swtch+0x141 cv_wait+0x70 ufs_thread_idle+0x147 thread_start+8 ffffff0007b58c40 SLEEP CV 1 swtch+0x141 cv_wait+0x70 ufs_thread_run+0x80 ufs_thread_hlock+0x73 thread_start+8 ffffff0255b72120 SLEEP CV 1 swtch+0x141 cv_wait+0x70 zio_wait+0x6b dmu_buf_hold_array_by_dnode+0x169 dmu_read_uio_dnode+0x5a dmu_read_uio_dbuf+0x51 zfs_read+0x1a3 fop_read+0xf3 pread+0x1a3 pread32+0x24 _sys_sysenter_post_swapgs+0x153 ffffff0007f1fc40 SLEEP CV 1 swtch+0x141 cv_wait+0x70 zio_wait+0x6b dmu_tx_count_write+0x645 dmu_tx_hold_write+0x5a zvol_strategy+0x1e5 bdev_strategy+0x61 spec_startio+0x8e spec_pageio+0x3a fop_pageio+0x59 swap_putapage+0x176 swap_putpage+0x295 fop_putpage+0x4c pageout+0x26a thread_start+8 ffffff0007df5c40 SLEEP CV 1 swtch+0x141 cv_wait+0x70 zio_wait+0x6b dsl_pool_sync_mos+0x4a dsl_pool_sync+0x25e spa_sync+0x386 txg_sync_thread+0x227 thread_start+8 ffffff024e4e6b40 SLEEP CV 1 swtch+0x141 cv_wait_sig+0x185 door_unref+0x94 doorfs32+0xf5 sys_syscall32+0x123 ffffff0251c124e0 SLEEP CV 1 swtch+0x141 cv_wait_sig+0x185 so_dequeue_msg+0x317 so_recvmsg+0x296 socket_recvmsg+0x33 socket_vop_read+0x5f fop_read+0xf3 read+0x2a7 read32+0x1e _sys_sysenter_post_swapgs+0x153 ffffff02516a1480 SLEEP CV 1 swtch+0x141 cv_wait_sig+0x185 str_cv_wait+0x27 strwaitq+0x2c3 strgetmsg+0x22b msgio32+0x23a getmsg32+0x98 _sys_sysenter_post_swapgs+0x153 ffffff024e518420 SLEEP CV 1 swtch+0x141 cv_wait_sig_swap_core+0x1b9 cv_wait_sig_swap+0x17 cv_waituntil_sig+0xbd port_getn+0x3cf portfs+0x25d portfs32+0x78 sys_syscall32+0x123 ffffff024e481040 SLEEP CV 1 swtch+0x141 cv_wait_sig_swap_core+0x1b9 cv_wait_sig_swap+0x17 cv_waituntil_sig+0xbd sigtimedwait+0x19c sys_syscall32+0x123 ffffff024e481b20 SLEEP CV 1 swtch+0x141 cv_wait_sig_swap_core+0x1b9 cv_wait_sig_swap+0x17 sigsuspend+0x101 sys_syscall32+0x123 ffffff0250d96b00 SLEEP CV 1 swtch+0x141 cv_wait_sig_swap_core+0x1b9 cv_wait_sig_swap+0x17 waitid+0x315 waitsys32+0x36 _sys_sysenter_post_swapgs+0x153 ffffff024e511b80 SLEEP SHUTTLE 1 swtch_to+0xb6 shuttle_resume+0x2af door_call+0x336 doorfs32+0xa7 sys_syscall32+0x123 ffffff000a913c40 RUN <NONE> 1 swtch+0x141 cv_timedwait_hires+0xec cv_reltimedwait+0x51 taskq_thread_wait+0x64 taskq_d_thread+0x145 thread_start+8 ffffff0007f25c40 RUN <NONE> 1 swtch+0x141 cv_wait+0x70 pageout_scanner+0x121 thread_start+8 ffffff0007e5cc40 ONPROC <NONE> 1 apix_intr_exit+0x24 apix_hilevel_intr_epilog+0xc1 apix_do_interrupt+0x34a _sys_rtt_ints_disabled+8 mutex_delay_default+7 apix_do_interrupt+0xfe _interrupt+0xba mach_cpu_idle+6 cpu_idle+0x11a idle+0xa7 thread_start+8 ffffff0007d5cc40 STOPPED <NONE> 1 <no consistent stack found> ffffff0007a05c40 PANIC <NONE> 1 1
There's a lot of stuff in page_create_throttle()
and friends. If we look at ::memstat
and ::kmastat
we can see the bleak picture painted for us.
> ::memstat Page Summary Pages MB %Tot ------------ ---------------- ---------------- ---- Kernel 861908 3366 82% Boot pages 68306 266 7% ZFS File Data 92003 359 9% Anon 22249 86 2% Exec and libs 193 0 0% Page cache 1471 5 0% Free (cachelist) 0 0 0% Free (freelist) 3 0 0% Total 1046133 4086 Physical 1046132 4086 > ::kmastat cache buf buf buf memory alloc alloc name size in use total in use succeed fail ------------------------------ ----- --------- --------- ------ ---------- ----- kmem_magazine_1 16 509 6275 100K 35328 0 kmem_magazine_3 32 368 3125 100K 6650 0 kmem_magazine_7 64 290 2046 132K 11255 0 kmem_magazine_15 128 233 2728 352K 19811 0 kmem_magazine_31 256 64 465 124K 1381 0 kmem_magazine_47 384 0 20 8K 89 0 kmem_magazine_63 512 0 21 12K 101 0 kmem_magazine_95 768 8 30 24K 203 0 kmem_magazine_143 1.12K 0 0 0 0 0 kmem_slab_cache 72 25453 35640 2.53M 54323 0 kmem_bufctl_cache 24 97454 97695 2.29M 132298 0 kmem_bufctl_audit_cache 192 0 0 0 0 0 kmem_va_4096 4K 16738 16768 65.5M 30623 0 kmem_va_8192 8K 15703 15712 123M 15805 0 kmem_va_12288 12K 322 330 4.12M 326 0 kmem_va_16384 16K 274 464 7.25M 1458 0 kmem_va_20480 20K 21 24 512K 23 0 kmem_va_24576 24K 24 25 640K 25 0 kmem_va_28672 28K 14 16 512K 15 0 kmem_va_32768 32K 74 76 2.38M 81 0 kmem_alloc_8 8 41165 51306 408K 828042 0 kmem_alloc_16 16 11051 15562 248K 202270 0 kmem_alloc_24 24 53920 60120 1.41M 990041 0 kmem_alloc_32 32 25369 28250 904K 933843 0 kmem_alloc_40 40 51362 51500 2.01M 253719 0 kmem_alloc_48 48 3385 9047 436K 606957 0 kmem_alloc_56 56 3673 3834 216K 37055 0 kmem_alloc_64 64 34161 35154 2.21M 842001 0 kmem_alloc_80 80 5583 10100 808K 156509 0 kmem_alloc_96 96 762 861 84K 70310 0 kmem_alloc_112 112 62100 62125 6.93M 350970 0 kmem_alloc_128 128 444 1116 144K 291126 0 kmem_alloc_160 160 1440 1700 272K 29369 0 kmem_alloc_192 192 452 660 132K 105399 0 kmem_alloc_224 224 736 799 188K 32597 0 kmem_alloc_256 256 15661 16470 4.29M 463054 0 kmem_alloc_320 320 45139 45168 14.7M 126745 0 kmem_alloc_384 384 976 1220 488K 17612 0 kmem_alloc_448 448 129 144 72K 4178 0 kmem_alloc_512 512 272 288 144K 10649 0 kmem_alloc_640 640 3028 3072 2M 34977 0 kmem_alloc_768 768 59 80 64K 2143 0 kmem_alloc_896 896 362 378 336K 5488 0 kmem_alloc_1152 1.12K 676 679 776K 66696 0 kmem_alloc_1344 1.31K 47 51 68K 568 0 kmem_alloc_1600 1.56K 42 50 80K 1743 0 kmem_alloc_2048 2K 494 546 1.07M 12084 0 kmem_alloc_2688 2.62K 131 150 400K 3077 0 kmem_alloc_4096 4K 574 579 2.26M 32223 0 kmem_alloc_8192 8K 146 153 1.20M 5437 0 kmem_alloc_12288 12K 281 287 3.36M 5428 0 kmem_alloc_16384 16K 32 38 608K 6299 0 kmem_alloc_24576 24K 16 19 456K 78 0 kmem_alloc_32768 32K 64 68 2.12M 1751 0 kmem_alloc_40960 40K 2 6 240K 20 0 kmem_alloc_49152 48K 0 5 240K 17 0 kmem_alloc_57344 56K 1 7 392K 22 0 kmem_alloc_65536 64K 8 11 704K 20 0 kmem_alloc_73728 72K 2 4 288K 8 0 kmem_alloc_81920 80K 0 3 240K 9 0 kmem_alloc_90112 88K 0 3 264K 4 0 kmem_alloc_98304 96K 2 4 384K 8 0 kmem_alloc_106496 104K 0 4 416K 9 0 kmem_alloc_114688 112K 0 4 448K 5 0 kmem_alloc_122880 120K 2 4 480K 5 0 kmem_alloc_131072 128K 8 15 1.88M 48834 0 streams_mblk 64 44414 44516 2.80M 130888 0 streams_dblk_16 128 59 93 12K 5805 0 streams_dblk_80 192 241 560 112K 18797 0 streams_dblk_144 256 2 45 12K 115701 0 streams_dblk_208 320 204 432 144K 4229 0 streams_dblk_272 384 0 50 20K 3976 0 streams_dblk_336 448 1 24 12K 149 0 streams_dblk_528 640 0 96 64K 38550 0 streams_dblk_1040 1.12K 0 35 40K 200 0 streams_dblk_1488 1.56K 0 10 16K 33 0 streams_dblk_1936 2K 0 18 36K 4611 0 streams_dblk_2576 2.62K 1 12 32K 121 0 streams_dblk_3856 3.88K 0 3 12K 9 0 streams_dblk_8192 112 0 31 4K 31 0 streams_dblk_12048 11.9K 0 3 36K 18 0 streams_dblk_16384 112 0 31 4K 2 0 streams_dblk_20240 19.9K 0 0 0 0 0 streams_dblk_24576 112 0 0 0 0 0 streams_dblk_28432 27.9K 0 0 0 0 0 streams_dblk_32768 112 0 0 0 0 0 streams_dblk_36624 35.9K 0 0 0 0 0 streams_dblk_40960 112 0 31 4K 1 0 streams_dblk_44816 43.9K 0 1 44K 1 0 streams_dblk_49152 112 0 0 0 0 0 streams_dblk_53008 51.9K 0 1 52K 1 0 streams_dblk_57344 112 0 31 4K 1 0 streams_dblk_61200 59.9K 0 0 0 0 0 streams_dblk_65536 112 0 0 0 0 0 streams_dblk_69392 67.9K 0 1 68K 1 0 streams_dblk_73728 112 0 31 4K 1 0 streams_dblk_esb 112 43532 43617 5.50M 328187 146 streams_fthdr 408 0 0 0 0 0 streams_ftblk 376 0 0 0 0 0 multidata 248 0 0 0 0 0 multidata_pdslab 6.95K 0 0 0 0 0 multidata_pattbl 32 0 0 0 0 0 log_cons_cache 48 11 83 4K 21 0 taskq_ent_cache 56 1302 1562 88K 40374 0 taskq_cache 280 135 154 44K 480 0 kmem_io_64G_128 128 0 0 0 0 0 kmem_io_64G_256 256 0 0 0 0 0 kmem_io_64G_512 512 0 0 0 0 0 kmem_io_64G_1024 1K 0 0 0 0 0 kmem_io_64G_2048 2K 0 0 0 0 0 kmem_io_64G_4096 4K 1024 1024 4M 1024 0 kmem_io_4G_128 128 0 0 0 0 0 kmem_io_4G_256 256 1 15 4K 18 0 kmem_io_4G_512 512 1 8 4K 2 0 kmem_io_4G_1024 1K 0 0 0 0 0 kmem_io_4G_2048 2K 0 0 0 0 0 kmem_io_4G_4096 4K 0 0 0 0 0 kmem_io_2G_128 128 0 0 0 0 0 kmem_io_2G_256 256 0 0 0 0 0 kmem_io_2G_512 512 0 0 0 0 0 kmem_io_2G_1024 1K 0 0 0 0 0 kmem_io_2G_2048 2K 0 0 0 0 0 kmem_io_2G_4096 4K 0 0 0 0 0 kmem_io_16M_128 128 0 0 0 0 0 kmem_io_16M_256 256 0 0 0 0 0 kmem_io_16M_512 512 0 0 0 0 0 kmem_io_16M_1024 1K 0 0 0 0 0 kmem_io_16M_2048 2K 0 0 0 0 0 kmem_io_16M_4096 4K 0 0 0 0 0 id32_cache 32 0 0 0 0 0 bp_map_4096 4K 0 0 0 0 0 bp_map_8192 8K 0 16 128K 11901 0 bp_map_12288 12K 0 10 128K 2 0 bp_map_16384 16K 0 0 0 0 0 bp_map_20480 20K 0 6 128K 2 0 bp_map_24576 24K 0 0 0 0 0 bp_map_28672 28K 0 0 0 0 0 bp_map_32768 32K 0 0 0 0 0 htable_t 72 2996 3025 220K 40624 0 hment_t 64 14727 28210 1.78M 1906098 0 hat_t 176 56 110 20K 5212 0 HatHash 8K 2 7 56K 767 0 HatVlpHash 4K 54 57 228K 5716 0 zfs_file_data_4096 4K 13 32 128K 42 0 zfs_file_data_8192 8K 719 720 5.62M 2795 0 zfs_file_data_12288 12K 10 20 256K 26 0 zfs_file_data_16384 16K 2 8 128K 4 0 zfs_file_data_20480 20K 14 30 640K 35 0 zfs_file_data_24576 24K 5 10 256K 6 0 zfs_file_data_28672 28K 10 16 512K 15 0 zfs_file_data_32768 32K 3 20 640K 85 0 segkp_4096 4K 53 64 256K 3335 0 segkp_8192 8K 0 0 0 0 0 segkp_12288 12K 0 0 0 0 0 segkp_16384 16K 0 0 0 0 0 segkp_20480 20K 0 0 0 0 0 umem_np_4096 4K 0 32 128K 165 0 umem_np_8192 8K 0 16 128K 68 0 umem_np_12288 12K 0 0 0 0 0 umem_np_16384 16K 0 8 128K 83 0 umem_np_20480 20K 0 6 128K 6 0 umem_np_24576 24K 0 5 128K 4 0 umem_np_28672 28K 0 0 0 0 0 umem_np_32768 32K 0 4 128K 68 0 mod_hash_entries 24 279 501 12K 11079 0 ipp_mod 304 0 0 0 0 0 ipp_action 368 0 0 0 0 0 ipp_packet 64 0 0 0 0 0 seg_cache 96 3070 3198 312K 214631 0 seg_pcache 104 0 38 4K 49 0 fnode_cache 176 3 20 4K 23 0 pipe_cache 320 13 36 12K 2056 0 snode_cache 152 252 286 44K 10744 0 dv_node_cache 176 50 242 44K 510 0 mac_impl_cache 13.4K 1 8 108K 1 0 mac_ring_cache 192 0 0 0 0 0 flow_entry_cache 26.7K 5 7 188K 7 0 flow_tab_cache 216 1 18 4K 1 0 mac_soft_ring_cache 376 3 10 4K 9 0 mac_srs_cache 3.16K 2 5 16K 6 0 mac_bcast_grp_cache 80 4 50 4K 6 0 mac_client_impl_cache 1.36K 1 8 12K 1 0 mac_promisc_impl_cache 120 0 0 0 0 0 dls_link_cache 352 1 11 4K 1 0 dls_devnet_cache 368 1 10 4K 1 0 sdev_node_cache 272 551 560 160K 1317 0 sdev_plugin 144 9 27 4K 9 0 dev_info_node_cache 680 141 156 104K 1198 0 ndi_fm_entry_cache 32 0 0 0 0 0 thread_cache 920 274 568 568K 5329 0 lwp_cache 1.73K 733 756 1.31M 1161 0 turnstile_cache 64 1018 2170 140K 9798 0 tslabel_cache 48 2 83 4K 2 0 cred_cache 200 137 180 36K 12136 0 rctl_cache 48 1007 1162 56K 47972 0 rctl_val_cache 64 1909 2108 136K 103117 0 task_cache 160 43 50 8K 94 0 kmem_defrag_cache 208 2 19 4K 2 0 kmem_move_cache 56 0 0 0 0 0 rootnex_dmahdl 2.53K 45743 45753 119M 54478 0 ddi_periodic 168 0 0 0 0 0 cyclic_id_cache 72 24 55 4K 32 0 callout_cachebabecafe 80 402 403 52K 402 0 callout_lcachebabecafe 48 727 744 48K 727 0 dnlc_space_cache 24 0 0 0 0 0 vfs_cache 208 47 57 12K 48 0 vn_cache 216 6398 6675 1.74M 16896 0 vsk_anchor_cache 40 20 100 4K 23 0 file_cache 64 423 558 36K 57519 0 stream_head_cache 376 128 150 60K 4832 0 queue_cache 656 310 330 220K 5786 0 syncq_cache 160 7 25 4K 25 0 qband_cache 64 2 62 4K 2 0 linkinfo_cache 48 4 83 4K 5 0 ciputctrl_cache 256 0 0 0 0 0 serializer_cache 64 23 62 4K 90 0 as_cache 224 55 102 24K 5211 0 marker_cache 120 0 33 4K 8252 0 anon_cache 48 23902 24485 1.15M 439375 0 anonmap_cache 112 2196 2275 260K 134741 0 segvn_cache 168 3015 3128 544K 195224 0 segvn_szc_cache1 4K 0 0 0 0 0 flk_edges 48 0 0 0 0 0 fdb_cache 104 0 0 0 0 0 timer_cache 144 2 27 4K 2 0 vmu_bound_cache 56 0 0 0 0 0 vmu_object_cache 64 0 0 0 0 0 physio_buf_cache 248 0 16 4K 90 0 process_cache 3.92K 60 66 264K 3745 0 ufs_inode_cache 368 2590 2590 1.01M 2590 0 directio_buf_cache 272 0 0 0 0 0 lufs_save 24 0 0 0 0 0 lufs_bufs 256 0 0 0 0 0 lufs_mapentry_cache 112 0 0 0 0 0 bootfs_node_cache 256 1 15 4K 1 0 audit_proc 40 61 100 4K 3353 0 drv_secobj_cache 296 0 0 0 0 0 dld_str_cache 304 3 13 4K 11 0 kcf_sreq_cache 56 0 0 0 0 0 kcf_areq_cache 296 0 0 0 0 0 kcf_context_cache 112 0 0 0 0 0 ip_minor_arena_sa_1 1 32 64 64 190 0 ip_minor_arena_la_1 1 7 64 64 123 0 ip_conn_cache 720 0 5 4K 2 0 tcp_conn_cache 1.77K 26 30 60K 65 0 udp_conn_cache 1.23K 32 39 52K 370 0 rawip_conn_cache 1.09K 0 0 0 0 0 rts_conn_cache 816 4 9 8K 9 0 ire_cache 352 18 33 12K 59 0 ncec_cache 200 3 20 4K 39 0 nce_cache 88 3 45 4K 47 0 rt_entry 152 10 26 4K 51 0 radix_mask 32 3 125 4K 5 0 radix_node 120 2 33 4K 2 0 ipsec_actions 72 0 0 0 0 0 ipsec_selectors 80 0 0 0 0 0 ipsec_policy 80 0 0 0 0 0 tcp_timercache 88 37 90 8K 245 0 tcp_notsack_blk_cache 24 0 0 0 0 0 squeue_cache 168 9 20 4K 9 0 sctp_conn_cache 2.47K 0 0 0 0 0 sctp_faddr_cache 176 0 0 0 0 0 sctp_set_cache 24 0 0 0 0 0 sctp_ftsn_set_cache 16 0 0 0 0 0 dce_cache 152 4 26 4K 4 0 ire_gw_secattr_cache 24 0 0 0 0 0 socket_cache 656 30 42 28K 250 0 socktpi_cache 960 0 4 4K 1 0 socktpi_unix_cache 960 15 32 32K 77 0 sock_sod_cache 648 0 0 0 0 0 exacct_object_cache 40 0 0 0 0 0 kssl_cache 1.59K 0 0 0 0 0 clnt_clts_endpnt_cache 88 0 0 0 0 0 range_seg_cache 64 1346 1488 96K 5765 0 zio_cache 936 28 344 344K 287859 0 zio_link_cache 48 16 2656 128K 143617 0 zio_buf_512 512 850 864 432K 29100 0 zio_data_buf_512 512 0 24 12K 386 0 zio_buf_1024 1K 222 368 368K 7984 0 zio_data_buf_1024 1K 0 8 8K 109 0 zio_buf_1536 1.50K 11 40 60K 255 0 zio_data_buf_1536 1.50K 0 8 12K 99 0 zio_buf_2048 2K 32 48 96K 1038 0 zio_data_buf_2048 2K 0 8 16K 106 0 zio_buf_2560 2.50K 22 40 100K 298 0 zio_data_buf_2560 2.50K 0 24 60K 108 0 zio_buf_3072 3K 17 32 96K 174 0 zio_data_buf_3072 3K 0 16 48K 81 0 zio_buf_3584 3.50K 25 32 112K 113 0 zio_data_buf_3584 3.50K 0 16 56K 76 0 zio_buf_4096 4K 15 20 80K 1730 0 zio_data_buf_4096 4K 0 4 16K 67 0 zio_buf_5120 5K 8 16 80K 300 0 zio_data_buf_5120 5K 0 8 40K 126 0 zio_buf_6144 6K 17 24 144K 150 0 zio_data_buf_6144 6K 0 4 24K 54 0 zio_buf_7168 7K 1 8 56K 90 0 zio_data_buf_7168 7K 0 8 56K 40 0 zio_buf_8192 8K 1 6 48K 165 0 zio_data_buf_8192 8K 719 719 5.62M 3178 0 zio_buf_10240 10K 7 14 140K 142 0 zio_data_buf_10240 10K 0 8 80K 43 0 zio_buf_12288 12K 1 6 72K 116 0 zio_data_buf_12288 12K 0 3 36K 23 0 zio_buf_14336 14K 1 6 84K 75 0 zio_data_buf_14336 14K 0 4 56K 19 0 zio_buf_16384 16K 140 151 2.36M 5221 0 zio_data_buf_16384 16K 0 2 32K 20 0 zio_buf_20480 20K 1 5 100K 60 0 zio_data_buf_20480 20K 1 5 100K 30 0 zio_buf_24576 24K 0 5 120K 27 0 zio_data_buf_24576 24K 1 5 120K 29 0 zio_buf_28672 28K 3 5 140K 19 0 zio_data_buf_28672 28K 0 4 112K 31 0 zio_buf_32768 32K 1 6 192K 17 0 zio_data_buf_32768 32K 0 3 96K 107 0 zio_buf_40960 40K 2 8 320K 51 0 zio_data_buf_40960 40K 1 7 280K 395 0 zio_buf_49152 48K 18 18 864K 39 0 zio_data_buf_49152 48K 3 6 288K 286 0 zio_buf_57344 56K 0 3 168K 4 0 zio_data_buf_57344 56K 29 31 1.70M 253 0 zio_buf_65536 64K 0 2 128K 7 0 zio_data_buf_65536 64K 609 611 38.2M 3598 0 zio_buf_81920 80K 0 6 480K 26 0 zio_data_buf_81920 80K 82 85 6.64M 112 0 zio_buf_98304 96K 0 7 672K 52 0 zio_data_buf_98304 96K 0 3 288K 9 0 zio_buf_114688 112K 0 6 672K 243 0 zio_data_buf_114688 112K 0 3 336K 4 0 zio_buf_131072 128K 5 13 1.62M 9607 0 zio_data_buf_131072 128K 2434 2441 305M 51825 0 zio_buf_163840 160K 0 0 0 0 0 zio_data_buf_163840 160K 0 0 0 0 0 zio_buf_196608 192K 0 0 0 0 0 zio_data_buf_196608 192K 0 0 0 0 0 zio_buf_229376 224K 0 0 0 0 0 zio_data_buf_229376 224K 0 0 0 0 0 zio_buf_262144 256K 0 0 0 0 0 zio_data_buf_262144 256K 0 0 0 0 0 zio_buf_327680 320K 0 0 0 0 0 zio_data_buf_327680 320K 0 0 0 0 0 zio_buf_393216 384K 0 0 0 0 0 zio_data_buf_393216 384K 0 0 0 0 0 zio_buf_458752 448K 0 0 0 0 0 zio_data_buf_458752 448K 0 0 0 0 0 zio_buf_524288 512K 0 0 0 0 0 zio_data_buf_524288 512K 0 0 0 0 0 zio_buf_655360 640K 0 0 0 0 0 zio_data_buf_655360 640K 0 0 0 0 0 zio_buf_786432 768K 0 0 0 0 0 zio_data_buf_786432 768K 0 0 0 0 0 zio_buf_917504 896K 0 0 0 0 0 zio_data_buf_917504 896K 0 0 0 0 0 zio_buf_1048576 1M 0 0 0 0 0 zio_data_buf_1048576 1M 0 0 0 0 0 zio_buf_1310720 1.25M 0 0 0 0 0 zio_data_buf_1310720 1.25M 0 0 0 0 0 zio_buf_1572864 1.50M 0 0 0 0 0 zio_data_buf_1572864 1.50M 0 0 0 0 0 zio_buf_1835008 1.75M 0 0 0 0 0 zio_data_buf_1835008 1.75M 0 0 0 0 0 zio_buf_2097152 2M 0 0 0 0 0 zio_data_buf_2097152 2M 0 0 0 0 0 zio_buf_2621440 2.50M 0 0 0 0 0 zio_data_buf_2621440 2.50M 0 0 0 0 0 zio_buf_3145728 3M 0 0 0 0 0 zio_data_buf_3145728 3M 0 0 0 0 0 zio_buf_3670016 3.50M 0 0 0 0 0 zio_data_buf_3670016 3.50M 0 0 0 0 0 zio_buf_4194304 4M 0 0 0 0 0 zio_data_buf_4194304 4M 0 0 0 0 0 zio_buf_5242880 5M 0 0 0 0 0 zio_data_buf_5242880 5M 0 0 0 0 0 zio_buf_6291456 6M 0 0 0 0 0 zio_data_buf_6291456 6M 0 0 0 0 0 zio_buf_7340032 7M 0 0 0 0 0 zio_data_buf_7340032 7M 0 0 0 0 0 zio_buf_8388608 8M 0 0 0 0 0 zio_data_buf_8388608 8M 0 0 0 0 0 zio_buf_10485760 10M 0 0 0 0 0 zio_data_buf_10485760 10M 0 0 0 0 0 zio_buf_12582912 12M 0 0 0 0 0 zio_data_buf_12582912 12M 0 0 0 0 0 zio_buf_14680064 14M 0 0 0 0 0 zio_data_buf_14680064 14M 0 0 0 0 0 zio_buf_16777216 16M 0 0 0 0 0 zio_data_buf_16777216 16M 0 0 0 0 0 sa_cache 120 398 429 52K 777 0 dnode_t 624 950 962 592K 2806 0 arc_buf_hdr_t_full 176 4860 9394 1.67M 45564 0 arc_buf_hdr_t_l2only 88 0 0 0 0 0 arc_buf_t 32 1536 2375 76K 45635 0 dmu_buf_impl_t 200 2155 2980 596K 47949 0 zil_lwb_cache 192 2 40 8K 69 0 zfs_znode_cache 248 398 400 100K 777 0 tl_cache 432 33 54 24K 285 0 keysock_1 1 0 0 0 0 0 spdsock_1 1 0 64 64 1 0 namefs_inodes_1 1 21 64 64 21 0 port_cache 80 18 50 4K 72 0 lnode_cache 32 3 125 4K 11 0 softmac_cache 568 1 7 4K 1 0 softmac_upper_cache 232 0 0 0 0 0 dtrace_state_cache 16K 0 0 0 0 0 blkdev0_xfer 152 0 52 8K 4338 0 blkdev1_xfer 152 0 52 8K 28963 0 vioif0_rx 80 43747 43900 3.43M 304081 146 idm_buf_cache 240 0 0 0 0 0 idm_task_cache 2.38K 0 0 0 0 0 idm_tx_pdu_cache 464 0 0 0 0 0 idm_rx_pdu_cache 513 0 0 0 0 0 idm_128k_buf_cache 128K 0 0 0 0 0 gsqueue_cb_cache 24 0 0 0 0 0 gsqueue_cpu_cache 24 4 167 4K 4 0 squeue_set_cache 64 1 62 4K 1 0 frameio_cache 784 0 0 0 0 0 vnd_str_cache 1.41K 0 0 0 0 0 vnd_dev_cache 192 0 0 0 0 0 vnd_pnsd_cache 264 1 15 4K 1 0 pty_map 64 1 62 4K 1 0 sppptun_map 440 0 0 0 0 0 authkern_cache 72 3 55 4K 4 0 authnone_cache 72 0 0 0 0 0 authloopback_cache 72 0 0 0 0 0 authdes_cache_handle 80 0 0 0 0 0 rnode_cache 656 0 0 0 0 0 nfs_access_cache 56 0 0 0 0 0 client_handle_cache 32 0 0 0 0 0 rnode4_cache 968 0 0 0 0 0 svnode_cache 40 0 0 0 0 0 nfs4_access_cache 56 0 0 0 0 0 client_handle4_cache 32 0 0 0 0 0 nfs4_ace4vals_cache 48 0 0 0 0 0 nfs4_ace4_list_cache 264 0 0 0 0 0 NFS_idmap_cache 48 4 83 4K 4 0 nlm_host_cache 240 0 0 0 0 0 nlm_vhold_cache 48 0 0 0 0 0 nlm_rpch_cache 32 0 0 0 0 0 nfslog_small_rec 512 0 0 0 0 0 nfslog_medium_rec 8K 0 0 0 0 0 nfslog_large_rec 32K 0 0 0 0 0 exi_cache_handle 120 3 33 4K 3 0 nfs_xuio_cache 664 0 0 0 0 0 Hex0xffffff024cf2f488_minor_1 1 0 0 0 0 0 Hex0xffffff024cf2f490_minor_1 1 0 0 0 0 0 overlay_plugin_cache 72 0 0 0 0 0 overlay_target 88 0 0 0 0 0 overlay_entry 184 0 0 0 0 0 Client_entry_cache 736 1 11 8K 1 0 ClntIP_entry_cache 376 0 0 0 0 0 OpenOwner_entry_cache 560 1 7 4K 1 0 OpenStateID_entry_cache 264 32 45 12K 32 0 LockStateID_entry_cache 528 0 0 0 0 0 Lockowner_entry_cache 176 0 0 0 0 0 File_entry_cache 296 31 39 12K 31 0 DelegStateID_entry_cache 216 0 36 8K 22 0 audit_buffer 152 0 26 4K 94 0 ------------------------------ ----- --------- --------- ------ ---------- ----- Total [hat_memload] 1.99M 1946722 0 Total [kmem_msb] 5.65M 261441 0 Total [kmem_va] 204M 48356 0 Total [kmem_default] 211M 9615106 292 Total [kmem_io_64G] 4M 1024 0 Total [kmem_io_4G] 8K 20 0 Total [bp_map] 384K 11905 0 Total [umem_np] 768K 394 0 Total [zfs_file_data] 8.12M 3008 0 Total [zfs_file_data_buf] 359M 61214 0 Total [segkp] 256K 3335 0 Total [ip_minor_arena_sa] 64 190 0 Total [ip_minor_arena_la] 64 123 0 Total [spdsock] 64 1 0 Total [namefs_inodes] 64 21 0 ------------------------------ ----- --------- --------- ------ ---------- ----- vmem memory memory memory alloc alloc name in use total import succeed fail ------------------------------ --------- ---------- --------- ---------- ----- heap 3.15G 1014G 0 56891 0 vmem_metadata 19.0M 19.1M 19.1M 4652 0 vmem_seg 17.8M 17.8M 17.8M 4539 0 vmem_hash 1000K 1004K 1004K 49 0 vmem_vmem 288K 320K 284K 92 0 static 0 0 0 0 0 static_alloc 0 0 0 0 0 hat_memload 1.99M 1.99M 1.99M 564 0 kstat 617K 652K 588K 2324 0 kmem_metadata 6.88M 8.12M 8.12M 2109 0 kmem_msb 5.65M 5.65M 5.65M 1912 0 kmem_cache 369K 472K 472K 544 0 kmem_hash 764K 780K 780K 341 4 kmem_log 193K 196K 196K 9 0 kmem_firewall_va 48.6M 48.6M 48.6M 138 0 kmem_firewall 0 0 0 0 0 kmem_oversize 48.6M 48.6M 48.6M 138 0 mod_sysfile 1.43K 4K 4K 39 0 kmem_va 215M 215M 215M 3492 0 kmem_default 211M 211M 211M 42479 292 kmem_io_64G 2.69G 2.85G 2.85G 51050 0 kmem_io_4G 8K 8K 8K 414 0 kmem_io_2G 0 0 0 0 0 kmem_io_16M 0 0 0 0 0 bp_map 496K 496K 496K 363 0 umem_np 768K 768K 768K 78 0 ksyms 1.88M 2.09M 2.09M 257 0 ctf 1.17M 1.44M 1.44M 249 0 heap_core 1.39M 888M 0 46 0 heaptext 7.92M 64M 0 182 0 module_text 7.98M 8.15M 7.92M 256 0 id32 0 0 0 0 0 module_data 1.03M 1.69M 1.39M 354 0 logminor_space 27 256K 0 37 0 taskq_id_arena 99 2.00G 0 386 0 zfs_file_data 361M 5.99G 0 24647 0 zfs_file_data_buf 359M 359M 359M 26424 0 device 1.20M 1G 0 44 0 segkp 24.6M 2G 0 5525 0 mac_minor_ids 5 127K 0 6 0 rctl_ids 44 32.0K 0 44 0 zoneid_space 0 9.76K 0 0 0 taskid_space 43 977K 0 94 0 pool_ids 0 977K 0 0 0 contracts 47 2.00G 0 106 0 ddi_periodic 0 1023 0 0 0 bootfs_minors 1 2.00G 0 1 0 ip_minor_arena_sa 64 256K 0 1 0 ip_minor_arena_la 64 4.00G 0 1 0 lib_va_32 7.73M 1.99G 0 20 0 lport-instances 0 64K 0 0 0 rport-instances 0 64K 0 0 0 tl_minor_space 33 256K 0 256 0 keysock 0 4.00G 0 0 0 spdsock 64 4.00G 0 1 0 namefs_inodes 64 64K 0 1 0 lofi_id 1 64.0M 0 1 0 lib_va_64 14.0M 125T 0 53 0 dtrace 72 4.00G 0 15229 0 dtrace_minor 0 4.00G 0 0 0 vnd_minors 0 2.00G 0 0 0 logdmux_minor 0 256 0 0 0 ptms_minor 1 16 0 1 0 ipf_minor 1 4.00G 0 23 0 sppptun_minor 0 16 0 0 0 devfsadm_event_channel 1 101 0 1 0 devfsadm_event_channel 1 2 0 1 0 syseventconfd_door 0 101 0 0 0 syseventconfd_door 1 2 0 1 0 syseventd_channel 1 101 0 1 0 syseventd_channel 1 2 0 1 0 Hex0xffffff024cf2f488_minor 0 4.00G 0 0 0 Hex0xffffff024cf2f490_minor 0 4.00G 0 0 0 overlay_target_minors 1 2.00G 0 1 0 Client_id_space 1 128K 0 1 0 ClntIP_id_space 0 1M 0 0 0 OpenOwner_id_space 1 1M 0 1 0 OpenStateID_id_space 32 1M 0 32 0 LockStateID_id_space 0 1M 0 0 0 Lockowner_id_space 0 1M 0 0 0 DelegStateID_id_space 0 1M 0 22 0 ------------------------------ --------- ---------- --------- ---------- -----
Okay, so the kmem_io_64G has a large amount of memory in it. This seems high for a 4 GiB VM, so where is that memory. When I looked at this, I then turned my attention to the vioif driver, since it has an interrupt thread stack trying to allocate memory. All of the vioif descriptors come out of a single kmem_cache: vioif0_rx. There are 43900 entries here. If we look at the cache, it's allocating DMA memory for each one equal to its max payload, give or take a few bytes. So in this case 64k. If we do the math, 43900 entries times about 64k is about 2.8 GiB! In many cases our MTU is much less than that and only about 1500 bytes or 9k; however, keeping the maximum size here does simplify the virtio implementation.
Okay, so with this knowledge, the question is why does this happen. Well, if we look at the virtio code it initially will fill up its entire descriptor pool, nothing wrong with that. It's not actually that many worth in the queue originally. Usually this number is on the order of 32-256 entries. We're a lot more than that. What happens is that the virtio net driver has no notion of backpressure for rx descriptors. It basically will happily allocate memory until the system runs out of it and constantly loan them up. Instead, we should have a fixed amount of descriptors we'll loan up and then move on to reloading. Otherwise we basically always risk running out of kernel memory by receiving more data than we can process.
Related issues
Updated by Joshua M. Clulow over 3 years ago
- Related to Feature #11329: improved Virtio framework added
Updated by Electric Monk over 3 years ago
- Status changed from New to Closed
- % Done changed from 0 to 100
git commit f8296c60994fb27105f37ac6f75661e4a6bdbab7
commit f8296c60994fb27105f37ac6f75661e4a6bdbab7 Author: Joshua M. Clulow <josh@sysmgr.org> Date: 2019-08-16T17:37:47.000Z 11329 improved Virtio framework 10012 vioblk should not accept an all-zero serial number 7366 vioif happily creates rx descriptors until it consumes all memory Reviewed by: Robert Mustacchi <rm@joyent.com> Approved by: Richard Lowe <richlowe@richlowe.net>