illumos gate

If we call arc_read with ARC_FLAG_PREFETCH and a non-null done callback, we fail this assertion in arc_access:
panic message: assertion failed: refcount_count(&hdr->b_l1hdr.b_refcnt) 0 (0x1 0x0)

The problem is that arc_read does:

} else {
            /*
     * This block is in the ghost cache.
...
            if (done != NULL)
                add_reference(hdr, hash_lock, private);
            if (*arc_flags & ARC_FLAG_PREFETCH)
                hdr->b_flags |= ARC_FLAG_PREFETCH;
…
            arc_access(hdr, hash_lock);

and arc_access does:

} else if (hdr->b_l1hdr.b_state == arc_mfu_ghost) {
...
        if (HDR_PREFETCH(hdr)) {
            ASSERT0(refcount_count(&hdr->b_l1hdr.b_refcnt));

normally this assertion works because add_reference() removes ARC_FLAG_PREFETCH from b_flags if it’s the first reference, but in this case we are setting the flag after calling add_reference().

I'm sure if the assertion can just be removed, or if this will cause some problem, like the prefetch flag stays set as we add more references (since it’s already >0).

We don't use this combination of arguments today, so this isn't a pressing issue, but it's something that's easy to think would work and then get bitten (e.g. https://github.com/openzfs/openzfs/pull/65)