illumos gate

From Matt Ahrens's original bug report at delphix:

Every snapshot has a deadlist, which tracks which blocks it deleted. The
deleted block pointers are stored in a bpobj specified by the creation time of
the block. There is one bpobj for every snapshot that existed at the time the
snapshot was created. There are therefore O(n^2) bpobjs for each filesystem (n = the number of snapshots of that filesystem).

A bpobj, even when empty, takes up at least ~512 bytes in the MOS's dnode
object. The total space wasted for empty bpobjs can be considerable (~512MB
for n=1000). Additionally, they dilute the dnode object to the pessimal case
where each block contains only 1 or 2 dnodes that we care about (the
dsl_dataset_phys_t and dsl_deadlist_phys_t); the others are all empty bpobjs
which we rarely access. This decreases caching efficiency and eliminates
economies of scale (where we read one dnode and end up getting a neighbor that
we need for free) when reading these blocks.

However, when there are many snapshots, the common case is for the vast
majority of these bpobjs to be completely empty. This is because blocks which
are freed around the same time (and thus will be in a given snapshot's
deadlist) tend to have been allocated around the same time, and thus be entered
into just a handful of bpobjs.

Therefore, we can improve performance, cache efficiency, and space usage by
eliminating the empty bpobjs. When initially allocating the bpobj, we will
instead use the object number of a pool-wide dummy, empty bpobj. If we
eventually need to modify the bpobj, we will create a new one at that point.
Use of the dummy bpobj (rather than a sentinel object number) allows this
on-disk format change to be read-only compatible, and allows code paths that
deal only with reading the bpobj to work without needing a special case.