When the system is under memory pressure (freemem is below lotsfree), the pageout_scanner() begins scanning memory to look for pages we can push out to disk-based swap. The I/O requests to send dirty pages to disk so that they may then be freed is performed by pageout(). As the system is often nearly out of memory, it is critical that this mechanism be able to perform I/O without blocking to allocate, and the mechanism we have to try and ensure that this is possible is the pageout reserve pool (pageout_reserve).

Any routine that pageout() calls is able to access pages in the reserve pool, even when other threads would be throttled (see NOMEMWAIT(), KM_PUSHPAGE, page_create_throttle(), etc). When swapping to ZFS, this includes any memory (e.g., ARC buffers, range tree additions) allocated in the eventual zvol_strategy() call. Unfortunately it does not extend to the rest of the ZFS I/O pipeline (see zio_execute()) which will still be blocked when it inevitably requires additional pages that are not available. If the pipeline becomes blocked, pool I/O will generally stall; if this stall does not resolve due to I/O already in flight, the system will enter the low memory deadlock condition from which there is no return.

This condition appears to have been acknowledged, and piecemeal fixes have been made through ZFS; one need only look for use of the KM_PUSHPAGE flag. As we increasingly take patches from other systems, where swap to ZFS is at least less popular, if not simply unavailable, it would seem that we need to increase structurally the likelihood that ZFS allocations will dip into the reserve.

I would like to add a new thread flag, T_PUSHPAGE, which would grant the same status as being part proc_pageout process today. zio_execute() would set this flag before entering the pipeline, and remove it once complete. If we discover more asynchronous contexts in ZFS that can block pageout() in future, we can ensure those threads get the treatment as well. In order to reduce risk, I am not currently proposing that we use this thread in place of the proc_pageout (etc) checks in NOMEMWAIT(), nor am I proposing unwinding the explicit use of KM_PUSHPAGE in ZFS today.