illumos gate

Once arc_no_grow is set, whenever we read a block from disk, we evict a block
from memory to make room for it. We evict from the ARC list that we will be
adding to. The list is based on whether the block is data vs metadata. So if
we are reading metadata, we will evict metadata. Therefore, the split of data
vs metadata in the ARC will stay more or less static, all else being equal.
This may be very detrimental to performance, if the working set size of
metadata is more than the amount of metadata currently in the ARC.

We have seen this problem on an NFS server where "zfs list" may need to cache
~100MB of metadata, but is unable to. Therefore every "zfs list" must read data off disk which is very slow.

The fix we propose is to evict approximately the oldest block, regardless of it
is data vs metadata, while respecting the arc_meta_limit. To do this
efficiently, we will determine which (data vs metadata) list's oldest entry is
older, and evict from that list. However, if we are
over the arc_meta_limit, we will evict metadata.