illumos gate

We have a ZFS NFS fileserver with 128 GB of RAM that wound up with persistent very high overall kernel memory usage and a very small ARC for this amount of RAM (we saw ~20 GB for a week+). In mdb -k, ::kmastat reported:

> ::kmastat
cache                            buf       buf       buf memory      alloc alloc
name                            size    in use     total in use    succeed  fail
------------------------------ ----- --------- --------- ------ ---------- -----
[...]
kmem_alloc_131072               128K         6    613033  74.8G  196862991     0

What is happening here is that kmem does not release deallocated objects right away for most caches. Instead they are transfered to per-CPU magazine caches, then old per-CPU magazines are added to the cache's cache_full magazine list. These unused magazines are only actually freed up when kmem_depot_ws_reap() is called, which as far as I can tell mostly happens only when kmem_reap() is called under hard memory pressure. Unless you have strong, rapid spikes in memory usage, this happens very rarely. In particular, pressure from the ZFS ARC does not seem to do this. Rather than reclaiming space from this 74 GB of unused memory, our ZFS ARC throttled itself to very small sizes (relatively speaking in our case).

Presumably at some point our fileserver needed all of this memory to handle incoming NFS requests (probably due to issue #3783), but now that it no longer does the kernel holding onto this memory is not doing anyone any good.

This issue was brought up and discussed on illumos-devel; the gmane incarnation of the thread about it is here and contains a more extended discussion.