l2arc compression buffers "leak"
I'm creating this issue on behalf of Andriy Gapon <avg@FreeBSD.org> who initially reported it to openzfs developers and developed the patch so it don't get lost.
Initial FreeBSD discussion:
Discussion on OpenZFS mailing list begins here:
After adding several assertions, etc., he have discovered:
To recap: after reports from FreeBSD ZFS users, I was able to capture two types of l2arc temporary compressions buffer leaks.
One was in arc_release() called from dbuf_dirty().
The other was in arc_hdr_destroy() called from arc_write_done() in the ZIO_FLAG_IO_REWRITE case. In both case the mentioned calls happened concurrently with l2arc writes and the corresponding buffers were in ARC_L2_WRITING state.
An earlier version of this patch has already been successfully tested by the original reporters. The patch also survives HybridCluster testing without any leaks or triggered assertions.
An earlier version of the patch is reviewed by Saso Kiselkov <email@example.com> ; this version of patch is the same patch of Andriy Gapon <avg@FreeBSD.org>'s updated one after Matthew's comments  and applied against Illumos tree and compile-tested in an illumos build environment. I (Xin Li <delphij@FreeBSD.org>) have reviewed the patch too.
It's the same patch we are currently shipping with FreeNAS.