illumos gate

The dataset's "compressed" size is incorrect:

ds_phys->ds_referenced_bytes = 0x705c9d14000 (7TB)
    ds_phys->ds_compressed_bytes = 0x4383f172a00 (4.6TB)
    ds_phys->ds_uncompressed_bytes = 0xc54e3a53e00 (13TB)

The compressed_bytes is determined from BP_GET_PSIZE(), so I looked at some
level-0 block pointers with zdb, and they reported size 0x200P (512 bytes).
While it's true that the data compressed down to less than 512 bytes, we
actually needed 4k for it on disk, as that is the minimum allocation unit.

The problem is that if the blkptr's psize is not aligned to the sector size (aka ashift), asize will be > psize (even when not on RAIDZ). We credit compression with the unaligned size (psize) even though we actually used (allocated) more space for the block so that it is sector-aligned.

The fix is to round the psize up to a multiple of the sector size. Note that the "sector size" mentioned here is the minimum sector size of normal-class (i.e. non-log, non-cache) devices. So the problem could still occur if there are normal-class vdevs with different ashift, but that is an uncommon, and not recommended, pool configuration.