Project

General

Profile

Bug #7366

vioif happily creates rx descriptors until it consumes all memory

Added by Robert Mustacchi almost 3 years ago. Updated 9 days ago.

Status:
Closed
Priority:
High
Category:
driver - device drivers
Start date:
2016-09-09
Due date:
% Done:

100%

Estimated time:
Difficulty:
Hard
Tags:

Description

We debugged a case today where a VM with 4 GB of memory was hanging. First, let's look at ::stacks from this machine:

THREAD           STATE    SOBJ                COUNT
ffffff0007a11c40 SLEEP    CV                    580
                 swtch+0x141
                 cv_wait+0x70
                 taskq_thread_wait+0xbe
                 taskq_thread+0x37c
                 thread_start+8

ffffff025185bb40 SLEEP    CV                     55
                 swtch+0x141
                 cv_wait_sig_swap_core+0x1b9
                 cv_wait_sig_swap+0x17
                 cv_waituntil_sig+0xbd
                 lwp_park+0x15e
                 syslwp_park+0x63
                 _sys_sysenter_post_swapgs+0x153

ffffff024e9467c0 SLEEP    CV                     29
                 swtch+0x141
                 cv_wait+0x70
                 page_create_throttle+0x16a
                 page_create_va+0x5d4
                 pvn_read_kluster+0x10c
                 ufs_getpage_miss+0x107
                 ufs_getpage+0x8a6
                 fop_getpage+0x7e
                 segvn_fault+0xdfa
                 as_fault+0x312
                 pagefault+0x96
                 trap+0x30c

ffffff024eabeae0 SLEEP    SHUTTLE                28
                 swtch+0x141
                 shuttle_swtch+0x203
                 door_return+0x214
                 doorfs32+0x180
                 _sys_sysenter_post_swapgs+0x153

ffffff0007a9ec40 FREE     <NONE>                 27
                 <no consistent stack found>

ffffff024e5dc480 SLEEP    CV                     20
                 swtch+0x141
                 cv_timedwait_sig_hires+0x39d
                 cv_waituntil_sig+0xfa
                 nanosleep+0x19f
                 _sys_sysenter_post_swapgs+0x153

ffffff0007cd5c40 SLEEP    CV                     19
                 swtch+0x141
                 cv_wait+0x70
                 evch_delivery_hold+0x70
                 evch_delivery_thr+0x2c4
                 thread_start+8

ffffff02515347e0 SLEEP    CV                     17
                 swtch+0x141
                 cv_wait_sig_swap_core+0x1b9
                 cv_wait_sig_swap+0x17
                 cv_timedwait_sig_hrtime+0x35
                 poll_common+0x288
                 pollsys+0x2dc
                 _sys_sysenter_post_swapgs+0x153

ffffff024ea3f0e0 SLEEP    SHUTTLE                15
                 swtch_to+0xb6
                 shuttle_resume+0x2af
                 door_return+0x3e0
                 doorfs32+0x180
                 _sys_sysenter_post_swapgs+0x153

ffffff0250f443e0 SLEEP    MUTEX                  14
                 swtch+0x141
                 turnstile_block+0x21a
                 mutex_vector_enter+0x3a3
                 txg_hold_open+0x55
                 dmu_tx_try_assign+0x87
                 dmu_tx_assign+0x60
                 zfs_write+0x4a6
                 fop_write+0xf3
                 do_io+0x9e
                 rfs4_op_write+0x433
                 rfs4_compound+0x2ed
                 rfs4_dispatch+0x264
                 common_dispatch+0xb2f
                 rfs_dispatch+0x2d
                 svc_getreq+0x1c1
                 svc_run+0xe0
                 svc_do_run+0x8e
                 nfssys+0x111
                 _sys_sysenter_post_swapgs+0x153

ffffff024e54b8a0 SLEEP    SHUTTLE                13
                 swtch_to+0xb6
                 shuttle_resume+0x2af
                 door_return+0x3e0
                 doorfs32+0x180
                 sys_syscall32+0x123

ffffff024ea3f820 SLEEP    CV                     12
                 swtch+0x141
                 cv_wait_sig_swap_core+0x1b9
                 cv_wait_sig_swap+0x17
                 pause+0x45
                 _sys_sysenter_post_swapgs+0x153

ffffff0007dc5c40 SLEEP    CV                      9
                 swtch+0x141
                 cv_wait+0x70
                 squeue_polling_thread+0xa9
                 thread_start+8

ffffff0008b62c40 SLEEP    CV                      9
                 swtch+0x141
                 cv_wait+0x70         
                 squeue_worker+0x114
                 thread_start+8

ffffff00089c8c40 SLEEP    CV                      8
                 swtch+0x141
                 cv_timedwait_hires+0xec
                 cv_timedwait_sig_hires+0x1eb
                 cv_reltimedwait_sig+0x4f
                 reaper_thread+0xbc
                 thread_start+8

ffffff02516993a0 SLEEP    CV                      8
                 <thread not in memory>

ffffff0007ad4c40 FREE     <NONE>                  6
                 av_dispatch_softvect+0x60
                 apix_dispatch_softint+0x41

ffffff000b2c5c40 SLEEP    CV                      6
                 swtch+0x141
                 cv_timedwait_hires+0xec
                 cv_reltimedwait+0x51
                 taskq_thread_wait+0x64
                 taskq_d_thread+0x145
                 thread_start+8

ffffff024e511440 SLEEP    CV                      6
                 swtch+0x141
                 cv_wait_sig_swap_core+0x1b9
                 cv_wait_sig_swap+0x17
                 cv_waituntil_sig+0xbd
                 lwp_park+0x15e
                 syslwp_park+0x63
                 sys_syscall32+0x123

ffffff025162d0a0 SLEEP    CV                      5
                 swtch+0x141
                 cv_timedwait_hires+0xec
                 cv_timedwait+0x5c
                 overlay_target_lookup_request+0x7e
                 overlay_target_ioctl+0x102
                 cdev_ioctl+0x39
                 spec_ioctl+0x60
                 fop_ioctl+0x55
                 ioctl+0x9b
                 _sys_sysenter_post_swapgs+0x153

ffffff0251af4820 SLEEP    CV                      5
                 swtch+0x141
                 cv_wait_sig_swap_core+0x1b9
                 cv_wait_sig_swap+0x17
                 sigsuspend+0x101
                 _sys_sysenter_post_swapgs+0x153

ffffff000c56cc40 RUN      <NONE>                  5
                 swtch+0x141
                 cv_wait+0x70         
                 taskq_thread_wait+0xbe
                 taskq_thread+0x37c
                 thread_start+8

ffffff0008d35c40 SLEEP    CV                      4
                 swtch+0x141
                 cv_timedwait_hires+0xec
                 cv_reltimedwait+0x51
                 kcfpool_svc+0x84
                 thread_start+8

ffffff025185b400 SLEEP    CV                      4
                 swtch+0x141
                 cv_wait_sig_swap_core+0x1b9
                 cv_wait_sig_swap+0x17
                 cv_waituntil_sig+0xbd
                 sigtimedwait+0x19c
                 _sys_sysenter_post_swapgs+0x153

ffffff024e5a8440 SLEEP    SHUTTLE                 4
                 swtch+0x141
                 shuttle_swtch+0x203
                 door_return+0x214
                 doorfs32+0x180
                 sys_syscall32+0x123

ffffff0008ba4c40 SLEEP    CV                      3
                 swtch+0x141
                 cv_wait+0x70
                 mac_soft_ring_worker+0xb1
                 thread_start+8

ffffff02570930a0 SLEEP    CV                      3
                 swtch+0x141
                 cv_wait+0x70
                 page_create_throttle+0x16a
                 page_create_va+0x5d4
                 swap_getapage+0x113
                 pvn_getpages+0x161
                 swap_getpage+0x61
                 fop_getpage+0x7e
                 anon_getpage+0x18d
                 segvn_faultpage+0x168
                 segvn_fault+0x8e6
                 as_fault+0x312
                 pagefault+0x96
                 trap+0x30c

ffffff0007ce1c40 ONPROC   <NONE>                  3
                 swtch+0x141
                 cpu_pause+0x80
                 thread_start+8

ffffff024e9a1b80 SLEEP    CV                      2
                 swtch+0x141
                 cv_timedwait_hires+0xec
                 txg_delay+0xd3       
                 dsl_dir_tempreserve_space+0x150
                 dmu_tx_try_assign+0x212
                 dmu_tx_assign+0x60
                 zfs_write+0x4a6
                 fop_write+0xf3
                 do_io+0x9e
                 rfs4_op_write+0x433
                 rfs4_compound+0x2ed
                 rfs4_dispatch+0x264
                 common_dispatch+0xb2f
                 rfs_dispatch+0x2d
                 svc_getreq+0x1c1
                 svc_run+0xe0
                 svc_do_run+0x8e
                 nfssys+0x111
                 _sys_sysenter_post_swapgs+0x153

ffffff0256109b20 SLEEP    CV                      2
                 swtch+0x141
                 cv_timedwait_sig_hires+0x39d
                 cv_timedwait_sig_hrtime+0x2a
                 poll_common+0xea
                 pollsys+0x2dc
                 _sys_sysenter_post_swapgs+0x153

ffffff024ea8fc40 SLEEP    CV                      2
                 swtch+0x141
                 cv_timedwait_sig_hires+0x39d
                 cv_timedwait_sig_hrtime+0x2a
                 poll_common+0x288
                 pollsys+0x2dc
                 _sys_sysenter_post_swapgs+0x153

ffffff024e5a87e0 SLEEP    CV                      2
                 swtch+0x141
                 cv_timedwait_sig_hires+0x39d
                 cv_waituntil_sig+0xfa
                 lwp_park+0x15e
                 syslwp_park+0x63
                 _sys_sysenter_post_swapgs+0x153

ffffff0008209c40 SLEEP    CV                      2
                 swtch+0x141
                 cv_wait+0x70
                 mac_srs_worker+0x141
                 thread_start+8

ffffff0007c0fc40 SLEEP    CV                      2
                 swtch+0x141
                 cv_wait+0x70
                 progressbar_key_abort_thread+0x73
                 thread_start+8

ffffff000c372c40 SLEEP    CV                      2
                 swtch+0x141
                 cv_wait+0x70
                 spa_thread+0x1db     
                 thread_start+8

ffffff000b2e7c40 SLEEP    CV                      2
                 swtch+0x141
                 cv_wait+0x70
                 svc_thread_creator+0xd8
                 thread_start+8

ffffff0007e2fc40 SLEEP    CV                      2
                 swtch+0x141
                 cv_wait+0x70
                 txg_thread_wait+0xaf
                 txg_quiesce_thread+0x106
                 thread_start+8

ffffff024e481780 SLEEP    CV                      2
                 swtch+0x141
                 cv_wait_sig+0x185
                 cte_get_event+0xb3
                 ctfs_endpoint_ioctl+0xf9
                 ctfs_bu_ioctl+0x4b
                 fop_ioctl+0x55
                 ioctl+0x9b
                 sys_syscall32+0x123

ffffff024e5fac40 SLEEP    CV                      2
                 swtch+0x141
                 cv_wait_sig+0x185
                 door_unref+0x94
                 doorfs32+0xf5
                 _sys_sysenter_post_swapgs+0x153

ffffff024e50d0c0 SLEEP    CV                      2
                 swtch+0x141
                 cv_wait_sig+0x185
                 svc_wait+0xb7
                 nfssys+0xd1
                 _sys_sysenter_post_swapgs+0x153

ffffff0256f1cc40 SLEEP    SHUTTLE                 2
                 swtch_to+0xb6
                 shuttle_resume+0x2af
                 door_call+0x336
                 doorfs32+0xa7
                 _sys_sysenter_post_swapgs+0x153

ffffff0007cedc40 ONPROC   <NONE>                  2
                 apix_intr_exit+0x24
                 apix_hilevel_intr_epilog+0xc1
                 apix_do_interrupt+0x34a
                 _sys_rtt_ints_disabled+8
                 default_lock_delay+0x50
                 apix_do_interrupt+0xfe
                 _interrupt+0xba
                 mach_cpu_idle+6
                 cpu_idle+0x11a
                 idle+0xa7            
                 thread_start+8

ffffff0007d3ec40 FREE     <NONE>                  1
                 apic_send_ipi+0x73
                 send_dirint+0x18
                 cbe_xcall+0xac
                 cyclic_reprogram_here+0x46
                 cyclic_reprogram+0x68
                 lbolt_ev_to_cyclic+0xcf
                 av_dispatch_softvect+0x60
                 apix_dispatch_softint+0x41

ffffff0007e43c40 FREE     <NONE>                  1
                 checked_rdmsr+0x42
                 ntv_rdmsr_xc+0x8c
                 call_func_ntv+0x130
                 ntv_rdmsr+0x3f
                 0xffffff024d2e3c48
                 gcpu_mca_logout+0x28a
                 gcpu_ntv_mca_poll+0x1fc
                 0xffffff024ca6fd40
                 cyclic_reprogram_here+0x46
                 apix_post_hardint+0x34
                 0xffffff024d1d6000
                 apix_dispatch_pending_hardint+0x3f

ffffff00083abc40 SLEEP    CV                      1
                 swtch+0x141
                 cv_timedwait_hires+0xec
                 arc_reclaim_thread+0x153
                 thread_start+8

ffffff000b8b0c40 SLEEP    CV                      1
                 swtch+0x141
                 cv_timedwait_hires+0xec
                 cv_reltimedwait+0x51
                 idm_wd_thread+0x203
                 thread_start+8

ffffff0007b8ec40 SLEEP    CV                      1
                 swtch+0x141
                 cv_timedwait_hires+0xec
                 cv_reltimedwait+0x51
                 kcfpoold+0xf6
                 thread_start+8

ffffff0007b4cc40 SLEEP    CV                      1
                 swtch+0x141
                 cv_timedwait_hires+0xec
                 cv_reltimedwait+0x51
                 page_capture_thread+0xb1
                 thread_start+8

ffffff0007f37c40 SLEEP    CV                      1
                 swtch+0x141
                 cv_timedwait_hires+0xec
                 cv_reltimedwait+0x51 
                 seg_pasync_thread+0xd1
                 thread_start+8

ffffff00083a5c40 SLEEP    CV                      1
                 swtch+0x141
                 cv_timedwait_hires+0xec
                 cv_reltimedwait+0x51
                 stmf_svc_timeout+0x112
                 stmf_svc+0x1c0
                 taskq_thread+0x2d0
                 thread_start+8

ffffff0007bd9c40 SLEEP    CV                      1
                 swtch+0x141
                 cv_timedwait_hires+0xec
                 cv_timedwait+0x5c
                 dce_reclaim_worker+0xab
                 thread_start+8

ffffff00083c8c40 SLEEP    CV                      1
                 swtch+0x141
                 cv_timedwait_hires+0xec
                 cv_timedwait+0x5c
                 l2arc_feed_thread+0xad
                 thread_start+8

ffffff000899ec40 SLEEP    CV                      1
                 swtch+0x141
                 cv_timedwait_hires+0xec
                 cv_timedwait+0x5c
                 nlm_gc+0x5c
                 thread_start+8

ffffff000b3eac40 SLEEP    CV                      1
                 swtch+0x141
                 cv_timedwait_hires+0xec
                 cv_timedwait+0x5c
                 txg_thread_wait+0x5f
                 txg_sync_thread+0x111
                 thread_start+8

ffffff00083b7c40 SLEEP    CV                      1
                 swtch+0x141
                 cv_timedwait_hires+0xec
                 dbuf_evict_thread+0xef
                 thread_start+8

ffffff024d2d4b00 SLEEP    CV                      1
                 swtch+0x141
                 cv_timedwait_sig_hires+0x39d
                 cv_timedwait_sig_hrtime+0x2a
                 poll_common+0x288
                 pollsys+0x2dc
                 sys_syscall32+0x123

ffffff024ea38100 SLEEP    CV                      1
                 swtch+0x141          
                 cv_timedwait_sig_hires+0x39d
                 cv_waituntil_sig+0xfa
                 lwp_park+0x15e
                 syslwp_park+0x63
                 sys_syscall32+0x123

ffffff0255b62020 SLEEP    CV                      1
                 swtch+0x141
                 cv_timedwait_sig_hires+0x39d
                 cv_waituntil_sig+0xfa
                 nanosleep+0x19f

ffffff0007fecc40 SLEEP    CV                      1
                 swtch+0x141
                 cv_wait+0x70
                 au_output_thread+0x80
                 taskq_thread+0x2d0
                 thread_start+8

ffffff000b3f0c40 SLEEP    CV                      1
                 swtch+0x141
                 cv_wait+0x70
                 crypto_bufcall_service+0x8d
                 thread_start+8

fffffffffbc31d80 SLEEP    CV                      1
                 swtch+0x141
                 cv_wait+0x70
                 dbuf_read+0x1ff
                 dmu_tx_check_ioerr+0x76
                 dmu_tx_count_write+0x49c
                 dmu_tx_hold_write+0x5a
                 zvol_strategy+0x1e5
                 bdev_strategy+0x61
                 spec_startio+0x8e
                 spec_pageio+0x3a
                 fop_pageio+0x59
                 swap_putapage+0x176
                 swap_putpage+0x295
                 fop_putpage+0x4c
                 segkp_unlock+0x21b
                 segkp_fault+0xfe
                 swapout+0x269
                 sched+0x8f7
                 main+0x46c

ffffff0007b88c40 SLEEP    CV                      1
                 swtch+0x141
                 cv_wait+0x70
                 dld_taskq_dispatch+0x115
                 thread_start+8

ffffff0007f2bc40 SLEEP    CV                      1
                 swtch+0x141
                 cv_wait+0x70
                 fsflush+0x21d
                thread_start+8       

ffffff000b8a4c40 SLEEP    CV                      1
                 swtch+0x141
                 cv_wait+0x70
                 i_mac_notify_thread+0xee
                 thread_start+8

ffffff0007bdfc40 SLEEP    CV                      1
                 swtch+0x141
                 cv_wait+0x70
                 ill_taskq_dispatch+0x173
                 thread_start+8

ffffff0007bc7c40 SLEEP    CV                      1
                 swtch+0x141
                 cv_wait+0x70
                 ipsec_loader+0x149
                 thread_start+8

ffffff0008f40c40 SLEEP    CV                      1
                 swtch+0x141
                 cv_wait+0x70
                 irm_balance_thread+0x50
                 thread_start+8

ffffff0008d7dc40 SLEEP    CV                      1
                 swtch+0x141
                 cv_wait+0x70
                 log_event_deliver+0x1b3
                 thread_start+8

ffffff0007f31c40 SLEEP    CV                      1
                 swtch+0x141
                 cv_wait+0x70
                 mod_uninstall_daemon+0x123
                 thread_start+8

ffffff0008984c40 SLEEP    CV                      1
                 swtch+0x141
                 cv_wait+0x70
                 nfsauth_refresh_thread+0x205
                 thread_start+8

ffffff0251534b80 SLEEP    CV                      1
                 swtch+0x141
                 cv_wait+0x70
                 page_create_throttle+0x16a
                 page_create_va+0x5d4
                 swap_getapage+0x113
                 pvn_getpages+0x161
                 swap_getpage+0x61
                 fop_getpage+0x7e
                 anon_zero+0xb6
                 segvn_lockop+0xd1a
                 as_ctl+0x2c5
                 as_map_locked+0x22b
                 as_map+0x58          
                 brk_internal+0x2b0
                 brk+0x51
                 _sys_sysenter_post_swapgs+0x153

ffffff0007ae6c40 SLEEP    CV                      1
                 swtch+0x141
                 cv_wait+0x70
                 page_create_wait+0x137
                 page_create_io+0x85
                 page_create_io_wrapper+0x47
                 segkmem_xalloc+0x8b
                 segkmem_alloc_io_64G+0x32
                 vmem_xalloc+0x5b1
                 vmem_alloc+0x135
                 kalloca+0x181
                 i_ddi_mem_alloc+0x15d
                 ddi_dma_mem_alloc+0x205
                 vioif_rx_construct+0xb0
                 kmem_cache_alloc+0x26b
                 vioif_add_rx+0x10d
                 vioif_populate_rx+0x1c
                 vioif_rx_handler+0x2d
                 apix_dispatch_pending_autovect+0x101
                 apix_dispatch_pending_hardint+0x34

ffffff0007b5ec40 SLEEP    CV                      1
                 swtch+0x141
                 cv_wait+0x70
                 pm_dep_thread+0xbd
                 thread_start+8

ffffff0007a23c40 SLEEP    CV                      1
                 swtch+0x141
                 cv_wait+0x70
                 scsi_hba_barrier_daemon+0xd6
                 thread_start+8

ffffff0007a29c40 SLEEP    CV                      1
                 swtch+0x141
                 cv_wait+0x70
                 scsi_lunchg1_daemon+0x1f4
                 thread_start+8

ffffff0007a2fc40 SLEEP    CV                      1
                 swtch+0x141
                 cv_wait+0x70
                 scsi_lunchg2_daemon+0x121
                 thread_start+8

ffffff000b2d5c40 SLEEP    CV                      1
                 swtch+0x141
                 cv_wait+0x70
                 softmac_taskq_dispatch+0x11d
                 thread_start+8

ffffff0007b3ac40 SLEEP    CV                      1
                 swtch+0x141          
                 cv_wait+0x70
                 streams_bufcall_service+0x8d
                 thread_start+8

ffffff0007b40c40 SLEEP    CV                      1
                 swtch+0x141
                 cv_wait+0x70
                 streams_qbkgrnd_service+0x151
                 thread_start+8

ffffff0007b46c40 SLEEP    CV                      1
                 swtch+0x141
                 cv_wait+0x70
                 streams_sqbkgrnd_service+0xe5
                 thread_start+8

ffffff0007ccfc40 SLEEP    CV                      1
                 swtch+0x141
                 cv_wait+0x70
                 task_commit+0xd9
                 thread_start+8

ffffff0007a0bc40 SLEEP    CV                      1
                 swtch+0x141
                 cv_wait+0x70
                 thread_reaper+0xb9
                 thread_start+8

ffffff0007b52c40 SLEEP    CV                      1
                 swtch+0x141
                 cv_wait+0x70
                 ufs_thread_idle+0x147
                 thread_start+8

ffffff0007b58c40 SLEEP    CV                      1
                 swtch+0x141
                 cv_wait+0x70
                 ufs_thread_run+0x80
                 ufs_thread_hlock+0x73
                 thread_start+8

ffffff0255b72120 SLEEP    CV                      1
                 swtch+0x141
                 cv_wait+0x70
                 zio_wait+0x6b
                 dmu_buf_hold_array_by_dnode+0x169
                 dmu_read_uio_dnode+0x5a
                 dmu_read_uio_dbuf+0x51
                 zfs_read+0x1a3
                 fop_read+0xf3
                 pread+0x1a3
                 pread32+0x24
                 _sys_sysenter_post_swapgs+0x153

ffffff0007f1fc40 SLEEP    CV                      1
                 swtch+0x141
                 cv_wait+0x70         
                 zio_wait+0x6b
                 dmu_tx_count_write+0x645
                 dmu_tx_hold_write+0x5a
                 zvol_strategy+0x1e5
                 bdev_strategy+0x61
                 spec_startio+0x8e
                 spec_pageio+0x3a
                 fop_pageio+0x59
                 swap_putapage+0x176
                 swap_putpage+0x295
                 fop_putpage+0x4c
                 pageout+0x26a
                 thread_start+8

ffffff0007df5c40 SLEEP    CV                      1
                 swtch+0x141
                 cv_wait+0x70
                 zio_wait+0x6b
                 dsl_pool_sync_mos+0x4a
                 dsl_pool_sync+0x25e
                 spa_sync+0x386
                 txg_sync_thread+0x227
                 thread_start+8

ffffff024e4e6b40 SLEEP    CV                      1
                 swtch+0x141
                 cv_wait_sig+0x185
                 door_unref+0x94
                 doorfs32+0xf5
                 sys_syscall32+0x123

ffffff0251c124e0 SLEEP    CV                      1
                 swtch+0x141
                 cv_wait_sig+0x185
                 so_dequeue_msg+0x317
                 so_recvmsg+0x296
                 socket_recvmsg+0x33
                 socket_vop_read+0x5f
                 fop_read+0xf3
                 read+0x2a7
                 read32+0x1e
                 _sys_sysenter_post_swapgs+0x153

ffffff02516a1480 SLEEP    CV                      1
                 swtch+0x141
                 cv_wait_sig+0x185
                 str_cv_wait+0x27
                 strwaitq+0x2c3
                 strgetmsg+0x22b
                 msgio32+0x23a
                 getmsg32+0x98
                 _sys_sysenter_post_swapgs+0x153

ffffff024e518420 SLEEP    CV                      1
                 swtch+0x141
                 cv_wait_sig_swap_core+0x1b9
                 cv_wait_sig_swap+0x17
                 cv_waituntil_sig+0xbd
                 port_getn+0x3cf
                 portfs+0x25d
                 portfs32+0x78
                 sys_syscall32+0x123

ffffff024e481040 SLEEP    CV                      1
                 swtch+0x141
                 cv_wait_sig_swap_core+0x1b9
                 cv_wait_sig_swap+0x17
                 cv_waituntil_sig+0xbd
                 sigtimedwait+0x19c
                 sys_syscall32+0x123

ffffff024e481b20 SLEEP    CV                      1
                 swtch+0x141
                 cv_wait_sig_swap_core+0x1b9
                 cv_wait_sig_swap+0x17
                 sigsuspend+0x101
                 sys_syscall32+0x123

ffffff0250d96b00 SLEEP    CV                      1
                 swtch+0x141
                 cv_wait_sig_swap_core+0x1b9
                 cv_wait_sig_swap+0x17
                 waitid+0x315
                 waitsys32+0x36
                 _sys_sysenter_post_swapgs+0x153

ffffff024e511b80 SLEEP    SHUTTLE                 1
                 swtch_to+0xb6
                 shuttle_resume+0x2af
                 door_call+0x336
                 doorfs32+0xa7
                 sys_syscall32+0x123

ffffff000a913c40 RUN      <NONE>                  1
                 swtch+0x141
                 cv_timedwait_hires+0xec
                 cv_reltimedwait+0x51
                 taskq_thread_wait+0x64
                 taskq_d_thread+0x145
                 thread_start+8

ffffff0007f25c40 RUN      <NONE>                  1
                 swtch+0x141
                 cv_wait+0x70
                 pageout_scanner+0x121
                 thread_start+8

ffffff0007e5cc40 ONPROC   <NONE>                  1
                 apix_intr_exit+0x24
                 apix_hilevel_intr_epilog+0xc1
                 apix_do_interrupt+0x34a
                 _sys_rtt_ints_disabled+8
                 mutex_delay_default+7
                 apix_do_interrupt+0xfe
                 _interrupt+0xba
                 mach_cpu_idle+6
                 cpu_idle+0x11a
                 idle+0xa7
                 thread_start+8

ffffff0007d5cc40 STOPPED  <NONE>                  1
                 <no consistent stack found>

ffffff0007a05c40 PANIC    <NONE>                  1
                 1

There's a lot of stuff in page_create_throttle() and friends. If we look at ::memstat and ::kmastat we can see the bleak picture painted for us.

> ::memstat 
Page Summary                Pages                MB  %Tot
------------     ----------------  ----------------  ----
Kernel                     861908              3366   82%
Boot pages                  68306               266    7%
ZFS File Data               92003               359    9%
Anon                        22249                86    2%
Exec and libs                 193                 0    0%
Page cache                   1471                 5    0%
Free (cachelist)                0                 0    0%
Free (freelist)                 3                 0    0%

Total                     1046133              4086
Physical                  1046132              4086

> ::kmastat

cache                            buf       buf       buf memory      alloc alloc
name                            size    in use     total in use    succeed  fail
------------------------------ ----- --------- --------- ------ ---------- -----
kmem_magazine_1                   16       509      6275   100K      35328     0
kmem_magazine_3                   32       368      3125   100K       6650     0
kmem_magazine_7                   64       290      2046   132K      11255     0
kmem_magazine_15                 128       233      2728   352K      19811     0
kmem_magazine_31                 256        64       465   124K       1381     0
kmem_magazine_47                 384         0        20     8K         89     0
kmem_magazine_63                 512         0        21    12K        101     0
kmem_magazine_95                 768         8        30    24K        203     0
kmem_magazine_143              1.12K         0         0      0          0     0
kmem_slab_cache                   72     25453     35640  2.53M      54323     0
kmem_bufctl_cache                 24     97454     97695  2.29M     132298     0
kmem_bufctl_audit_cache          192         0         0      0          0     0
kmem_va_4096                      4K     16738     16768  65.5M      30623     0
kmem_va_8192                      8K     15703     15712   123M      15805     0
kmem_va_12288                    12K       322       330  4.12M        326     0
kmem_va_16384                    16K       274       464  7.25M       1458     0
kmem_va_20480                    20K        21        24   512K         23     0
kmem_va_24576                    24K        24        25   640K         25     0
kmem_va_28672                    28K        14        16   512K         15     0
kmem_va_32768                    32K        74        76  2.38M         81     0
kmem_alloc_8                       8     41165     51306   408K     828042     0
kmem_alloc_16                     16     11051     15562   248K     202270     0
kmem_alloc_24                     24     53920     60120  1.41M     990041     0
kmem_alloc_32                     32     25369     28250   904K     933843     0
kmem_alloc_40                     40     51362     51500  2.01M     253719     0
kmem_alloc_48                     48      3385      9047   436K     606957     0
kmem_alloc_56                     56      3673      3834   216K      37055     0
kmem_alloc_64                     64     34161     35154  2.21M     842001     0
kmem_alloc_80                     80      5583     10100   808K     156509     0
kmem_alloc_96                     96       762       861    84K      70310     0
kmem_alloc_112                   112     62100     62125  6.93M     350970     0
kmem_alloc_128                   128       444      1116   144K     291126     0
kmem_alloc_160                   160      1440      1700   272K      29369     0
kmem_alloc_192                   192       452       660   132K     105399     0
kmem_alloc_224                   224       736       799   188K      32597     0
kmem_alloc_256                   256     15661     16470  4.29M     463054     0
kmem_alloc_320                   320     45139     45168  14.7M     126745     0
kmem_alloc_384                   384       976      1220   488K      17612     0
kmem_alloc_448                   448       129       144    72K       4178     0
kmem_alloc_512                   512       272       288   144K      10649     0
kmem_alloc_640                   640      3028      3072     2M      34977     0
kmem_alloc_768                   768        59        80    64K       2143     0
kmem_alloc_896                   896       362       378   336K       5488     0
kmem_alloc_1152                1.12K       676       679   776K      66696     0
kmem_alloc_1344                1.31K        47        51    68K        568     0
kmem_alloc_1600                1.56K        42        50    80K       1743     0
kmem_alloc_2048                   2K       494       546  1.07M      12084     0
kmem_alloc_2688                2.62K       131       150   400K       3077     0
kmem_alloc_4096                   4K       574       579  2.26M      32223     0
kmem_alloc_8192                   8K       146       153  1.20M       5437     0
kmem_alloc_12288                 12K       281       287  3.36M       5428     0
kmem_alloc_16384                 16K        32        38   608K       6299     0
kmem_alloc_24576                 24K        16        19   456K         78     0
kmem_alloc_32768                 32K        64        68  2.12M       1751     0
kmem_alloc_40960                 40K         2         6   240K         20     0
kmem_alloc_49152                 48K         0         5   240K         17     0
kmem_alloc_57344                 56K         1         7   392K         22     0
kmem_alloc_65536                 64K         8        11   704K         20     0
kmem_alloc_73728                 72K         2         4   288K          8     0
kmem_alloc_81920                 80K         0         3   240K          9     0
kmem_alloc_90112                 88K         0         3   264K          4     0
kmem_alloc_98304                 96K         2         4   384K          8     0
kmem_alloc_106496               104K         0         4   416K          9     0
kmem_alloc_114688               112K         0         4   448K          5     0
kmem_alloc_122880               120K         2         4   480K          5     0
kmem_alloc_131072               128K         8        15  1.88M      48834     0
streams_mblk                      64     44414     44516  2.80M     130888     0
streams_dblk_16                  128        59        93    12K       5805     0
streams_dblk_80                  192       241       560   112K      18797     0
streams_dblk_144                 256         2        45    12K     115701     0
streams_dblk_208                 320       204       432   144K       4229     0
streams_dblk_272                 384         0        50    20K       3976     0
streams_dblk_336                 448         1        24    12K        149     0
streams_dblk_528                 640         0        96    64K      38550     0
streams_dblk_1040              1.12K         0        35    40K        200     0
streams_dblk_1488              1.56K         0        10    16K         33     0
streams_dblk_1936                 2K         0        18    36K       4611     0
streams_dblk_2576              2.62K         1        12    32K        121     0
streams_dblk_3856              3.88K         0         3    12K          9     0
streams_dblk_8192                112         0        31     4K         31     0
streams_dblk_12048             11.9K         0         3    36K         18     0
streams_dblk_16384               112         0        31     4K          2     0
streams_dblk_20240             19.9K         0         0      0          0     0
streams_dblk_24576               112         0         0      0          0     0
streams_dblk_28432             27.9K         0         0      0          0     0
streams_dblk_32768               112         0         0      0          0     0
streams_dblk_36624             35.9K         0         0      0          0     0
streams_dblk_40960               112         0        31     4K          1     0
streams_dblk_44816             43.9K         0         1    44K          1     0
streams_dblk_49152               112         0         0      0          0     0
streams_dblk_53008             51.9K         0         1    52K          1     0
streams_dblk_57344               112         0        31     4K          1     0
streams_dblk_61200             59.9K         0         0      0          0     0
streams_dblk_65536               112         0         0      0          0     0
streams_dblk_69392             67.9K         0         1    68K          1     0
streams_dblk_73728               112         0        31     4K          1     0
streams_dblk_esb                 112     43532     43617  5.50M     328187   146
streams_fthdr                    408         0         0      0          0     0
streams_ftblk                    376         0         0      0          0     0
multidata                        248         0         0      0          0     0
multidata_pdslab               6.95K         0         0      0          0     0
multidata_pattbl                  32         0         0      0          0     0
log_cons_cache                    48        11        83     4K         21     0
taskq_ent_cache                   56      1302      1562    88K      40374     0
taskq_cache                      280       135       154    44K        480     0
kmem_io_64G_128                  128         0         0      0          0     0
kmem_io_64G_256                  256         0         0      0          0     0
kmem_io_64G_512                  512         0         0      0          0     0
kmem_io_64G_1024                  1K         0         0      0          0     0
kmem_io_64G_2048                  2K         0         0      0          0     0
kmem_io_64G_4096                  4K      1024      1024     4M       1024     0
kmem_io_4G_128                   128         0         0      0          0     0
kmem_io_4G_256                   256         1        15     4K         18     0
kmem_io_4G_512                   512         1         8     4K          2     0
kmem_io_4G_1024                   1K         0         0      0          0     0
kmem_io_4G_2048                   2K         0         0      0          0     0
kmem_io_4G_4096                   4K         0         0      0          0     0
kmem_io_2G_128                   128         0         0      0          0     0
kmem_io_2G_256                   256         0         0      0          0     0
kmem_io_2G_512                   512         0         0      0          0     0
kmem_io_2G_1024                   1K         0         0      0          0     0
kmem_io_2G_2048                   2K         0         0      0          0     0
kmem_io_2G_4096                   4K         0         0      0          0     0
kmem_io_16M_128                  128         0         0      0          0     0
kmem_io_16M_256                  256         0         0      0          0     0
kmem_io_16M_512                  512         0         0      0          0     0
kmem_io_16M_1024                  1K         0         0      0          0     0
kmem_io_16M_2048                  2K         0         0      0          0     0
kmem_io_16M_4096                  4K         0         0      0          0     0
id32_cache                        32         0         0      0          0     0
bp_map_4096                       4K         0         0      0          0     0
bp_map_8192                       8K         0        16   128K      11901     0
bp_map_12288                     12K         0        10   128K          2     0
bp_map_16384                     16K         0         0      0          0     0
bp_map_20480                     20K         0         6   128K          2     0
bp_map_24576                     24K         0         0      0          0     0
bp_map_28672                     28K         0         0      0          0     0
bp_map_32768                     32K         0         0      0          0     0
htable_t                          72      2996      3025   220K      40624     0
hment_t                           64     14727     28210  1.78M    1906098     0
hat_t                            176        56       110    20K       5212     0
HatHash                           8K         2         7    56K        767     0
HatVlpHash                        4K        54        57   228K       5716     0
zfs_file_data_4096                4K        13        32   128K         42     0
zfs_file_data_8192                8K       719       720  5.62M       2795     0
zfs_file_data_12288              12K        10        20   256K         26     0
zfs_file_data_16384              16K         2         8   128K          4     0
zfs_file_data_20480              20K        14        30   640K         35     0
zfs_file_data_24576              24K         5        10   256K          6     0
zfs_file_data_28672              28K        10        16   512K         15     0
zfs_file_data_32768              32K         3        20   640K         85     0
segkp_4096                        4K        53        64   256K       3335     0
segkp_8192                        8K         0         0      0          0     0
segkp_12288                      12K         0         0      0          0     0
segkp_16384                      16K         0         0      0          0     0
segkp_20480                      20K         0         0      0          0     0
umem_np_4096                      4K         0        32   128K        165     0
umem_np_8192                      8K         0        16   128K         68     0
umem_np_12288                    12K         0         0      0          0     0
umem_np_16384                    16K         0         8   128K         83     0
umem_np_20480                    20K         0         6   128K          6     0
umem_np_24576                    24K         0         5   128K          4     0
umem_np_28672                    28K         0         0      0          0     0
umem_np_32768                    32K         0         4   128K         68     0
mod_hash_entries                  24       279       501    12K      11079     0
ipp_mod                          304         0         0      0          0     0
ipp_action                       368         0         0      0          0     0
ipp_packet                        64         0         0      0          0     0
seg_cache                         96      3070      3198   312K     214631     0
seg_pcache                       104         0        38     4K         49     0
fnode_cache                      176         3        20     4K         23     0
pipe_cache                       320        13        36    12K       2056     0
snode_cache                      152       252       286    44K      10744     0
dv_node_cache                    176        50       242    44K        510     0
mac_impl_cache                 13.4K         1         8   108K          1     0
mac_ring_cache                   192         0         0      0          0     0
flow_entry_cache               26.7K         5         7   188K          7     0
flow_tab_cache                   216         1        18     4K          1     0
mac_soft_ring_cache              376         3        10     4K          9     0
mac_srs_cache                  3.16K         2         5    16K          6     0
mac_bcast_grp_cache               80         4        50     4K          6     0
mac_client_impl_cache          1.36K         1         8    12K          1     0
mac_promisc_impl_cache           120         0         0      0          0     0
dls_link_cache                   352         1        11     4K          1     0
dls_devnet_cache                 368         1        10     4K          1     0
sdev_node_cache                  272       551       560   160K       1317     0
sdev_plugin                      144         9        27     4K          9     0
dev_info_node_cache              680       141       156   104K       1198     0
ndi_fm_entry_cache                32         0         0      0          0     0
thread_cache                     920       274       568   568K       5329     0
lwp_cache                      1.73K       733       756  1.31M       1161     0
turnstile_cache                   64      1018      2170   140K       9798     0
tslabel_cache                     48         2        83     4K          2     0
cred_cache                       200       137       180    36K      12136     0
rctl_cache                        48      1007      1162    56K      47972     0
rctl_val_cache                    64      1909      2108   136K     103117     0
task_cache                       160        43        50     8K         94     0
kmem_defrag_cache                208         2        19     4K          2     0
kmem_move_cache                   56         0         0      0          0     0
rootnex_dmahdl                 2.53K     45743     45753   119M      54478     0
ddi_periodic                     168         0         0      0          0     0
cyclic_id_cache                   72        24        55     4K         32     0
callout_cachebabecafe             80       402       403    52K        402     0
callout_lcachebabecafe            48       727       744    48K        727     0
dnlc_space_cache                  24         0         0      0          0     0
vfs_cache                        208        47        57    12K         48     0
vn_cache                         216      6398      6675  1.74M      16896     0
vsk_anchor_cache                  40        20       100     4K         23     0
file_cache                        64       423       558    36K      57519     0
stream_head_cache                376       128       150    60K       4832     0
queue_cache                      656       310       330   220K       5786     0
syncq_cache                      160         7        25     4K         25     0
qband_cache                       64         2        62     4K          2     0
linkinfo_cache                    48         4        83     4K          5     0
ciputctrl_cache                  256         0         0      0          0     0
serializer_cache                  64        23        62     4K         90     0
as_cache                         224        55       102    24K       5211     0
marker_cache                     120         0        33     4K       8252     0
anon_cache                        48     23902     24485  1.15M     439375     0
anonmap_cache                    112      2196      2275   260K     134741     0
segvn_cache                      168      3015      3128   544K     195224     0
segvn_szc_cache1                  4K         0         0      0          0     0
flk_edges                         48         0         0      0          0     0
fdb_cache                        104         0         0      0          0     0
timer_cache                      144         2        27     4K          2     0
vmu_bound_cache                   56         0         0      0          0     0
vmu_object_cache                  64         0         0      0          0     0
physio_buf_cache                 248         0        16     4K         90     0
process_cache                  3.92K        60        66   264K       3745     0
ufs_inode_cache                  368      2590      2590  1.01M       2590     0
directio_buf_cache               272         0         0      0          0     0
lufs_save                         24         0         0      0          0     0
lufs_bufs                        256         0         0      0          0     0
lufs_mapentry_cache              112         0         0      0          0     0
bootfs_node_cache                256         1        15     4K          1     0
audit_proc                        40        61       100     4K       3353     0
drv_secobj_cache                 296         0         0      0          0     0
dld_str_cache                    304         3        13     4K         11     0
kcf_sreq_cache                    56         0         0      0          0     0
kcf_areq_cache                   296         0         0      0          0     0
kcf_context_cache                112         0         0      0          0     0
ip_minor_arena_sa_1                1        32        64     64        190     0
ip_minor_arena_la_1                1         7        64     64        123     0
ip_conn_cache                    720         0         5     4K          2     0
tcp_conn_cache                 1.77K        26        30    60K         65     0
udp_conn_cache                 1.23K        32        39    52K        370     0
rawip_conn_cache               1.09K         0         0      0          0     0
rts_conn_cache                   816         4         9     8K          9     0
ire_cache                        352        18        33    12K         59     0
ncec_cache                       200         3        20     4K         39     0
nce_cache                         88         3        45     4K         47     0
rt_entry                         152        10        26     4K         51     0
radix_mask                        32         3       125     4K          5     0
radix_node                       120         2        33     4K          2     0
ipsec_actions                     72         0         0      0          0     0
ipsec_selectors                   80         0         0      0          0     0
ipsec_policy                      80         0         0      0          0     0
tcp_timercache                    88        37        90     8K        245     0
tcp_notsack_blk_cache             24         0         0      0          0     0
squeue_cache                     168         9        20     4K          9     0
sctp_conn_cache                2.47K         0         0      0          0     0
sctp_faddr_cache                 176         0         0      0          0     0
sctp_set_cache                    24         0         0      0          0     0
sctp_ftsn_set_cache               16         0         0      0          0     0
dce_cache                        152         4        26     4K          4     0
ire_gw_secattr_cache              24         0         0      0          0     0
socket_cache                     656        30        42    28K        250     0
socktpi_cache                    960         0         4     4K          1     0
socktpi_unix_cache               960        15        32    32K         77     0
sock_sod_cache                   648         0         0      0          0     0
exacct_object_cache               40         0         0      0          0     0
kssl_cache                     1.59K         0         0      0          0     0
clnt_clts_endpnt_cache            88         0         0      0          0     0
range_seg_cache                   64      1346      1488    96K       5765     0
zio_cache                        936        28       344   344K     287859     0
zio_link_cache                    48        16      2656   128K     143617     0
zio_buf_512                      512       850       864   432K      29100     0
zio_data_buf_512                 512         0        24    12K        386     0
zio_buf_1024                      1K       222       368   368K       7984     0
zio_data_buf_1024                 1K         0         8     8K        109     0
zio_buf_1536                   1.50K        11        40    60K        255     0
zio_data_buf_1536              1.50K         0         8    12K         99     0
zio_buf_2048                      2K        32        48    96K       1038     0
zio_data_buf_2048                 2K         0         8    16K        106     0
zio_buf_2560                   2.50K        22        40   100K        298     0
zio_data_buf_2560              2.50K         0        24    60K        108     0
zio_buf_3072                      3K        17        32    96K        174     0
zio_data_buf_3072                 3K         0        16    48K         81     0
zio_buf_3584                   3.50K        25        32   112K        113     0
zio_data_buf_3584              3.50K         0        16    56K         76     0
zio_buf_4096                      4K        15        20    80K       1730     0
zio_data_buf_4096                 4K         0         4    16K         67     0
zio_buf_5120                      5K         8        16    80K        300     0
zio_data_buf_5120                 5K         0         8    40K        126     0
zio_buf_6144                      6K        17        24   144K        150     0
zio_data_buf_6144                 6K         0         4    24K         54     0
zio_buf_7168                      7K         1         8    56K         90     0
zio_data_buf_7168                 7K         0         8    56K         40     0
zio_buf_8192                      8K         1         6    48K        165     0
zio_data_buf_8192                 8K       719       719  5.62M       3178     0
zio_buf_10240                    10K         7        14   140K        142     0
zio_data_buf_10240               10K         0         8    80K         43     0
zio_buf_12288                    12K         1         6    72K        116     0
zio_data_buf_12288               12K         0         3    36K         23     0
zio_buf_14336                    14K         1         6    84K         75     0
zio_data_buf_14336               14K         0         4    56K         19     0
zio_buf_16384                    16K       140       151  2.36M       5221     0
zio_data_buf_16384               16K         0         2    32K         20     0
zio_buf_20480                    20K         1         5   100K         60     0
zio_data_buf_20480               20K         1         5   100K         30     0
zio_buf_24576                    24K         0         5   120K         27     0
zio_data_buf_24576               24K         1         5   120K         29     0
zio_buf_28672                    28K         3         5   140K         19     0
zio_data_buf_28672               28K         0         4   112K         31     0
zio_buf_32768                    32K         1         6   192K         17     0
zio_data_buf_32768               32K         0         3    96K        107     0
zio_buf_40960                    40K         2         8   320K         51     0
zio_data_buf_40960               40K         1         7   280K        395     0
zio_buf_49152                    48K        18        18   864K         39     0
zio_data_buf_49152               48K         3         6   288K        286     0
zio_buf_57344                    56K         0         3   168K          4     0
zio_data_buf_57344               56K        29        31  1.70M        253     0
zio_buf_65536                    64K         0         2   128K          7     0
zio_data_buf_65536               64K       609       611  38.2M       3598     0
zio_buf_81920                    80K         0         6   480K         26     0
zio_data_buf_81920               80K        82        85  6.64M        112     0
zio_buf_98304                    96K         0         7   672K         52     0
zio_data_buf_98304               96K         0         3   288K          9     0
zio_buf_114688                  112K         0         6   672K        243     0
zio_data_buf_114688             112K         0         3   336K          4     0
zio_buf_131072                  128K         5        13  1.62M       9607     0
zio_data_buf_131072             128K      2434      2441   305M      51825     0
zio_buf_163840                  160K         0         0      0          0     0
zio_data_buf_163840             160K         0         0      0          0     0
zio_buf_196608                  192K         0         0      0          0     0
zio_data_buf_196608             192K         0         0      0          0     0
zio_buf_229376                  224K         0         0      0          0     0
zio_data_buf_229376             224K         0         0      0          0     0
zio_buf_262144                  256K         0         0      0          0     0
zio_data_buf_262144             256K         0         0      0          0     0
zio_buf_327680                  320K         0         0      0          0     0
zio_data_buf_327680             320K         0         0      0          0     0
zio_buf_393216                  384K         0         0      0          0     0
zio_data_buf_393216             384K         0         0      0          0     0
zio_buf_458752                  448K         0         0      0          0     0
zio_data_buf_458752             448K         0         0      0          0     0
zio_buf_524288                  512K         0         0      0          0     0
zio_data_buf_524288             512K         0         0      0          0     0
zio_buf_655360                  640K         0         0      0          0     0
zio_data_buf_655360             640K         0         0      0          0     0
zio_buf_786432                  768K         0         0      0          0     0
zio_data_buf_786432             768K         0         0      0          0     0
zio_buf_917504                  896K         0         0      0          0     0
zio_data_buf_917504             896K         0         0      0          0     0
zio_buf_1048576                   1M         0         0      0          0     0
zio_data_buf_1048576              1M         0         0      0          0     0
zio_buf_1310720                1.25M         0         0      0          0     0
zio_data_buf_1310720           1.25M         0         0      0          0     0
zio_buf_1572864                1.50M         0         0      0          0     0
zio_data_buf_1572864           1.50M         0         0      0          0     0
zio_buf_1835008                1.75M         0         0      0          0     0
zio_data_buf_1835008           1.75M         0         0      0          0     0
zio_buf_2097152                   2M         0         0      0          0     0
zio_data_buf_2097152              2M         0         0      0          0     0
zio_buf_2621440                2.50M         0         0      0          0     0
zio_data_buf_2621440           2.50M         0         0      0          0     0
zio_buf_3145728                   3M         0         0      0          0     0
zio_data_buf_3145728              3M         0         0      0          0     0
zio_buf_3670016                3.50M         0         0      0          0     0
zio_data_buf_3670016           3.50M         0         0      0          0     0
zio_buf_4194304                   4M         0         0      0          0     0
zio_data_buf_4194304              4M         0         0      0          0     0
zio_buf_5242880                   5M         0         0      0          0     0
zio_data_buf_5242880              5M         0         0      0          0     0
zio_buf_6291456                   6M         0         0      0          0     0
zio_data_buf_6291456              6M         0         0      0          0     0
zio_buf_7340032                   7M         0         0      0          0     0
zio_data_buf_7340032              7M         0         0      0          0     0
zio_buf_8388608                   8M         0         0      0          0     0
zio_data_buf_8388608              8M         0         0      0          0     0
zio_buf_10485760                 10M         0         0      0          0     0
zio_data_buf_10485760            10M         0         0      0          0     0
zio_buf_12582912                 12M         0         0      0          0     0
zio_data_buf_12582912            12M         0         0      0          0     0
zio_buf_14680064                 14M         0         0      0          0     0
zio_data_buf_14680064            14M         0         0      0          0     0
zio_buf_16777216                 16M         0         0      0          0     0
zio_data_buf_16777216            16M         0         0      0          0     0
sa_cache                         120       398       429    52K        777     0
dnode_t                          624       950       962   592K       2806     0
arc_buf_hdr_t_full               176      4860      9394  1.67M      45564     0
arc_buf_hdr_t_l2only              88         0         0      0          0     0
arc_buf_t                         32      1536      2375    76K      45635     0
dmu_buf_impl_t                   200      2155      2980   596K      47949     0
zil_lwb_cache                    192         2        40     8K         69     0
zfs_znode_cache                  248       398       400   100K        777     0
tl_cache                         432        33        54    24K        285     0
keysock_1                          1         0         0      0          0     0
spdsock_1                          1         0        64     64          1     0
namefs_inodes_1                    1        21        64     64         21     0
port_cache                        80        18        50     4K         72     0
lnode_cache                       32         3       125     4K         11     0
softmac_cache                    568         1         7     4K          1     0
softmac_upper_cache              232         0         0      0          0     0
dtrace_state_cache               16K         0         0      0          0     0
blkdev0_xfer                     152         0        52     8K       4338     0
blkdev1_xfer                     152         0        52     8K      28963     0
vioif0_rx                         80     43747     43900  3.43M     304081   146
idm_buf_cache                    240         0         0      0          0     0
idm_task_cache                 2.38K         0         0      0          0     0
idm_tx_pdu_cache                 464         0         0      0          0     0
idm_rx_pdu_cache                 513         0         0      0          0     0
idm_128k_buf_cache              128K         0         0      0          0     0
gsqueue_cb_cache                  24         0         0      0          0     0
gsqueue_cpu_cache                 24         4       167     4K          4     0
squeue_set_cache                  64         1        62     4K          1     0
frameio_cache                    784         0         0      0          0     0
vnd_str_cache                  1.41K         0         0      0          0     0
vnd_dev_cache                    192         0         0      0          0     0
vnd_pnsd_cache                   264         1        15     4K          1     0
pty_map                           64         1        62     4K          1     0
sppptun_map                      440         0         0      0          0     0
authkern_cache                    72         3        55     4K          4     0
authnone_cache                    72         0         0      0          0     0
authloopback_cache                72         0         0      0          0     0
authdes_cache_handle              80         0         0      0          0     0
rnode_cache                      656         0         0      0          0     0
nfs_access_cache                  56         0         0      0          0     0
client_handle_cache               32         0         0      0          0     0
rnode4_cache                     968         0         0      0          0     0
svnode_cache                      40         0         0      0          0     0
nfs4_access_cache                 56         0         0      0          0     0
client_handle4_cache              32         0         0      0          0     0
nfs4_ace4vals_cache               48         0         0      0          0     0
nfs4_ace4_list_cache             264         0         0      0          0     0
NFS_idmap_cache                   48         4        83     4K          4     0
nlm_host_cache                   240         0         0      0          0     0
nlm_vhold_cache                   48         0         0      0          0     0
nlm_rpch_cache                    32         0         0      0          0     0
nfslog_small_rec                 512         0         0      0          0     0
nfslog_medium_rec                 8K         0         0      0          0     0
nfslog_large_rec                 32K         0         0      0          0     0
exi_cache_handle                 120         3        33     4K          3     0
nfs_xuio_cache                   664         0         0      0          0     0
Hex0xffffff024cf2f488_minor_1      1         0         0      0          0     0
Hex0xffffff024cf2f490_minor_1      1         0         0      0          0     0
overlay_plugin_cache              72         0         0      0          0     0
overlay_target                    88         0         0      0          0     0
overlay_entry                    184         0         0      0          0     0
Client_entry_cache               736         1        11     8K          1     0
ClntIP_entry_cache               376         0         0      0          0     0
OpenOwner_entry_cache            560         1         7     4K          1     0
OpenStateID_entry_cache          264        32        45    12K         32     0
LockStateID_entry_cache          528         0         0      0          0     0
Lockowner_entry_cache            176         0         0      0          0     0
File_entry_cache                 296        31        39    12K         31     0
DelegStateID_entry_cache         216         0        36     8K         22     0
audit_buffer                     152         0        26     4K         94     0
------------------------------ ----- --------- --------- ------ ---------- -----
Total [hat_memload]                                       1.99M    1946722     0
Total [kmem_msb]                                          5.65M     261441     0
Total [kmem_va]                                            204M      48356     0
Total [kmem_default]                                       211M    9615106   292
Total [kmem_io_64G]                                          4M       1024     0
Total [kmem_io_4G]                                           8K         20     0
Total [bp_map]                                             384K      11905     0
Total [umem_np]                                            768K        394     0
Total [zfs_file_data]                                     8.12M       3008     0
Total [zfs_file_data_buf]                                  359M      61214     0
Total [segkp]                                              256K       3335     0
Total [ip_minor_arena_sa]                                    64        190     0
Total [ip_minor_arena_la]                                    64        123     0
Total [spdsock]                                              64          1     0
Total [namefs_inodes]                                        64         21     0
------------------------------ ----- --------- --------- ------ ---------- -----

vmem                              memory     memory    memory      alloc alloc 
name                              in use      total    import    succeed  fail 
------------------------------ --------- ---------- --------- ---------- ----- 
heap                               3.15G      1014G         0      56891     0 
    vmem_metadata                  19.0M      19.1M     19.1M       4652     0 
        vmem_seg                   17.8M      17.8M     17.8M       4539     0 
        vmem_hash                  1000K      1004K     1004K         49     0 
        vmem_vmem                   288K       320K      284K         92     0 
    static                             0          0         0          0     0 
        static_alloc                   0          0         0          0     0 
    hat_memload                    1.99M      1.99M     1.99M        564     0 
    kstat                           617K       652K      588K       2324     0 
    kmem_metadata                  6.88M      8.12M     8.12M       2109     0 
        kmem_msb                   5.65M      5.65M     5.65M       1912     0 
        kmem_cache                  369K       472K      472K        544     0 
        kmem_hash                   764K       780K      780K        341     4 
    kmem_log                        193K       196K      196K          9     0 
    kmem_firewall_va               48.6M      48.6M     48.6M        138     0 
        kmem_firewall                  0          0         0          0     0 
        kmem_oversize              48.6M      48.6M     48.6M        138     0 
    mod_sysfile                    1.43K         4K        4K         39     0 
    kmem_va                         215M       215M      215M       3492     0 
        kmem_default                211M       211M      211M      42479   292 
    kmem_io_64G                    2.69G      2.85G     2.85G      51050     0 
    kmem_io_4G                        8K         8K        8K        414     0 
    kmem_io_2G                         0          0         0          0     0 
    kmem_io_16M                        0          0         0          0     0 
    bp_map                          496K       496K      496K        363     0 
    umem_np                         768K       768K      768K         78     0 
    ksyms                          1.88M      2.09M     2.09M        257     0 
    ctf                            1.17M      1.44M     1.44M        249     0 
heap_core                          1.39M       888M         0         46     0 
    heaptext                       7.92M        64M         0        182     0 
        module_text                7.98M      8.15M     7.92M        256     0 
    id32                               0          0         0          0     0 
    module_data                    1.03M      1.69M     1.39M        354     0 
logminor_space                        27       256K         0         37     0 
taskq_id_arena                        99      2.00G         0        386     0 
zfs_file_data                       361M      5.99G         0      24647     0 
    zfs_file_data_buf               359M       359M      359M      26424     0 
device                             1.20M         1G         0         44     0 
segkp                              24.6M         2G         0       5525     0 
mac_minor_ids                          5       127K         0          6     0 
rctl_ids                              44      32.0K         0         44     0 
zoneid_space                           0      9.76K         0          0     0 
taskid_space                          43       977K         0         94     0 
pool_ids                               0       977K         0          0     0 
contracts                             47      2.00G         0        106     0 
ddi_periodic                           0       1023         0          0     0 
bootfs_minors                          1      2.00G         0          1     0 
ip_minor_arena_sa                     64       256K         0          1     0 
ip_minor_arena_la                     64      4.00G         0          1     0 
lib_va_32                          7.73M      1.99G         0         20     0 
lport-instances                        0        64K         0          0     0 
rport-instances                        0        64K         0          0     0 
tl_minor_space                        33       256K         0        256     0 
keysock                                0      4.00G         0          0     0 
spdsock                               64      4.00G         0          1     0 
namefs_inodes                         64        64K         0          1     0 
lofi_id                                1      64.0M         0          1     0 
lib_va_64                          14.0M       125T         0         53     0 
dtrace                                72      4.00G         0      15229     0 
dtrace_minor                           0      4.00G         0          0     0 
vnd_minors                             0      2.00G         0          0     0 
logdmux_minor                          0        256         0          0     0 
ptms_minor                             1         16         0          1     0 
ipf_minor                              1      4.00G         0         23     0 
sppptun_minor                          0         16         0          0     0 
devfsadm_event_channel                 1        101         0          1     0 
devfsadm_event_channel                 1          2         0          1     0 
syseventconfd_door                     0        101         0          0     0 
syseventconfd_door                     1          2         0          1     0 
syseventd_channel                      1        101         0          1     0 
syseventd_channel                      1          2         0          1     0 
Hex0xffffff024cf2f488_minor            0      4.00G         0          0     0 
Hex0xffffff024cf2f490_minor            0      4.00G         0          0     0 
overlay_target_minors                  1      2.00G         0          1     0 
Client_id_space                        1       128K         0          1     0 
ClntIP_id_space                        0         1M         0          0     0 
OpenOwner_id_space                     1         1M         0          1     0 
OpenStateID_id_space                  32         1M         0         32     0 
LockStateID_id_space                   0         1M         0          0     0 
Lockowner_id_space                     0         1M         0          0     0 
DelegStateID_id_space                  0         1M         0         22     0 
------------------------------ --------- ---------- --------- ---------- ----- 

Okay, so the kmem_io_64G has a large amount of memory in it. This seems high for a 4 GiB VM, so where is that memory. When I looked at this, I then turned my attention to the vioif driver, since it has an interrupt thread stack trying to allocate memory. All of the vioif descriptors come out of a single kmem_cache: vioif0_rx. There are 43900 entries here. If we look at the cache, it's allocating DMA memory for each one equal to its max payload, give or take a few bytes. So in this case 64k. If we do the math, 43900 entries times about 64k is about 2.8 GiB! In many cases our MTU is much less than that and only about 1500 bytes or 9k; however, keeping the maximum size here does simplify the virtio implementation.

Okay, so with this knowledge, the question is why does this happen. Well, if we look at the virtio code it initially will fill up its entire descriptor pool, nothing wrong with that. It's not actually that many worth in the queue originally. Usually this number is on the order of 32-256 entries. We're a lot more than that. What happens is that the virtio net driver has no notion of backpressure for rx descriptors. It basically will happily allocate memory until the system runs out of it and constantly loan them up. Instead, we should have a fixed amount of descriptors we'll loan up and then move on to reloading. Otherwise we basically always risk running out of kernel memory by receiving more data than we can process.


Related issues

Related to illumos gate - Feature #11329: improved Virtio frameworkClosed

Actions

History

#1

Updated by Joshua M. Clulow 4 months ago

  • Assignee set to Joshua M. Clulow
#2

Updated by Joshua M. Clulow 10 days ago

#3

Updated by Electric Monk 9 days ago

  • Status changed from New to Closed
  • % Done changed from 0 to 100

git commit f8296c60994fb27105f37ac6f75661e4a6bdbab7

commit  f8296c60994fb27105f37ac6f75661e4a6bdbab7
Author: Joshua M. Clulow <josh@sysmgr.org>
Date:   2019-08-16T17:37:47.000Z

    11329 improved Virtio framework
    10012 vioblk should not accept an all-zero serial number
    7366 vioif happily creates rx descriptors until it consumes all memory
    Reviewed by: Robert Mustacchi <rm@joyent.com>
    Approved by: Richard Lowe <richlowe@richlowe.net>

Also available in: Atom PDF