Bug #13096
closedxnf asleep at wheel while freemem smashes into the ground
100%
Description
The xnf
driver (for Xen virtual network interfaces) performs a variety of exciting allocations within xnf_send()
. When the system is under memory pressure, those allocations can block waiting for pages to become available. Also while the system is under memory pressure, various kmem caches need to be reaped to free up pages; at least one of those (ip_ire_reclaim()
) appears to need to execute in an squeue. The reap can get stuck waiting to enter the squeue forever, preventing any subsequent reaping and all but ensuring deadlock.
In a dump from the firing of the pageout deadman, we can see the kmem_taskq
is active and about half way through a reap:
> ::taskq -n kmem_taskq ADDR NAME ACT/THDS Q'ED MAXQ INST fffffe0bd3416c68 kmem_taskq 1/ 1 236 482 0
It is stuck in ip_ire_reclaim()
:
> ::taskq -n kmem_taskq | ::walk taskq_thread | ::stacks THREAD STATE SOBJ COUNT fffffe000f417c20 SLEEP CV 1 swtch+0x133 cv_wait+0x68 tcp_ixa_cleanup_wait_and_finish+0x53 conn_ixa_cleanup+0x1a3 ipcl_walk+0x91 ip_ire_reclaim_stack+0x56 ip_ire_reclaim+0x36 kmem_cache_reap+0x3e taskq_thread+0x2cd thread_start+0xb
The conn_ixa_cleanup()
routine kicks some work into an squeue and waits for it to complete. Where are we on squeues:
> ::squeue -v ADDR STATE CPU FIRST LAST WORKER fffffe0bd3d05d00 00329 1 fffffe0bdceede60 fffffe0bdecd81c0 fffffe0010bd7c20 | +--> SQS_PROC being processed SQS_FAST ... in fast-path mode SQS_BOUND worker thread bound to CPU ADDR STATE CPU FIRST LAST WORKER fffffe0bd3d05dc0 00820 1 0000000000000000 0000000000000000 fffffe000fcc5c20 | +--> SQS_BOUND worker thread bound to CPU ADDR STATE CPU FIRST LAST WORKER fffffe0bd3d05e80 00820 0 0000000000000000 0000000000000000 fffffe000f58bc20 | +--> SQS_BOUND worker thread bound to CPU
Ah, one is running. Let's see what it is doing:
> fffffe0bd3d05d00::print squeue_t sq_run | ::stacks THREAD STATE SOBJ COUNT fffffe0bdd9920c0 SLEEP CV 1 swtch+0x133 cv_wait+0x68 page_create_throttle+0x174 page_create_va+0x598 segkmem_page_create+0x97 segkmem_xalloc+0x13f segkmem_alloc_vn+0x3b segkmem_alloc+0x17 vmem_xalloc+0x629 vmem_alloc+0x190 kmem_slab_create+0x7c kmem_slab_alloc+0x10b kmem_cache_alloc+0x15b xnf_data_txbuf_alloc+0x17 xnf_mblk_map+0x69 xnf_send+0x23f mac_ring_tx+0x43 mac_provider_tx+0x85 mac_tx+0x295 str_mdata_fastpath_put+0x8e ip_xmit+0x843 ire_send_wire_v4+0x345 conn_ip_output+0x1d4 tcp_send_data+0x58 tcp_output+0x582 squeue_enter+0x3f9 tcp_sendmsg+0x16c so_sendmsg+0x24a socket_sendmsg+0x62 socket_vop_write+0x61 fop_write+0x60 write+0x2c6
Ah.
Updated by Joshua M. Clulow over 1 year ago
Testing Notes¶
I built an AMI build from OpenIndiana plus the RTI build bits. I ran iperf
between the test guest built from the new AMI, and an existing instance running other bits. Networking appeared stable, so I tried running the machine out of memory. If I killed the test early enough, after creating some but not terminally sustained pressure, the networking stack seemed to be able to continue to function. Because not all of the pageout fixes are in, after sustained paging activity the machine deadlocked, but the networking stuff did not appear to be the cause.
Updated by Joshua M. Clulow over 1 year ago
To fix this, I changed allocations for buffers in the data path from sleeping (KM_SLEEP
) to non-sleeping (KM_NOSLEEP
) so that they will ultimately fail and return NULL
when there are no pages readily available instead of blocking. The TX paths that use these allocation functions already have error return paths which appear relatively sound, so we will now exercise those paths on allocation failures as well. Initial allocations such as when the driver is starting up are still sleeping allocations.
Updated by Electric Monk over 1 year ago
- Status changed from New to Closed
- % Done changed from 0 to 100
git commit cfd17c15945080ff766acfba4bfbc0ac4d2d31cd
commit cfd17c15945080ff766acfba4bfbc0ac4d2d31cd Author: Joshua M. Clulow <josh@sysmgr.org> Date: 2020-09-23T22:22:51.000Z 13096 xnf asleep at wheel while freemem smashes into the ground Reviewed by: Jason King <jason.brian.king@gmail.com> Approved by: Robert Mustacchi <rm@fingolfin.org>