illumos gate

Original analysis by Matt Ahrens:

The problem is that when a dataset is last closed (e.g. a filesystem is
unmounted), we need to get rid of all its state in the DMU (dbufs, dnodes,
etc).  This needs to be coordinated with the ARC (because the arc bufs have
pointers to these data structures too).  We end up evicting all the data out of
the ARC.

This results in very bad performance for things like "zfs get all" if some
filesystems/zvols/snapshots are not mounted (or for zvols, are not open /
exported by comstar), because every snapshot's objset will be opened (to get
the zpl/zvol props), read, and then closed (which will be the last close,
evicting all the data from the ARC).  When the next snapshot is opened, it
probably shares all its relevant blocks with the previous snapshot, but we just
explicitly evicted all of those blocks from the ARC.  So we end up repeatedly
reading in the same blocks from disk.

The fix is to change the interaction between the DMU and ARC so that when the
DMU is shutting down an objset, we do not evict the data from the ARC.  Instead
we simply coordinate the destruction of the DMU's data with the ARC.  The only
case where we actually need to explicitly evict from the ARC is when
dbuf_rele_and_unlock() determines that the administrator has requested that it
not be kept in memory, via the primarycache/secondarycache properties.  In this
case, we evict the data from the ARC by its blkptr_t, the same way as when a
block is freed we explicitly evict it from the ARC.