Kernel kmem can hold significant amounts of unused memory in cache cache_full magazines
We have a ZFS NFS fileserver with 128 GB of RAM that wound up with persistent very high overall kernel memory usage and a very small ARC for this amount of RAM (we saw ~20 GB for a week+). In mdb -k, ::kmastat reported:
> ::kmastat cache buf buf buf memory alloc alloc name size in use total in use succeed fail ------------------------------ ----- --------- --------- ------ ---------- ----- [...] kmem_alloc_131072 128K 6 613033 74.8G 196862991 0
What is happening here is that kmem does not release deallocated objects right away for most caches. Instead they are transfered to per-CPU magazine caches, then old per-CPU magazines are added to the cache's cache_full magazine list. These unused magazines are only actually freed up when kmem_depot_ws_reap() is called, which as far as I can tell mostly happens only when kmem_reap() is called under hard memory pressure. Unless you have strong, rapid spikes in memory usage, this happens very rarely. In particular, pressure from the ZFS ARC does not seem to do this. Rather than reclaiming space from this 74 GB of unused memory, our ZFS ARC throttled itself to very small sizes (relatively speaking in our case).
Presumably at some point our fileserver needed all of this memory to handle incoming NFS requests (probably due to issue #3783), but now that it no longer does the kernel holding onto this memory is not doing anyone any good.
This issue was brought up and discussed on illumos-devel; the gmane incarnation of the thread about it is here and contains a more extended discussion.
No data to display