illumos gate

This is a description based on my incomplete knowledge of ZFS send/receive internals. When updates for sparse files are sent, the stream contains "free"-items that get processed by restore_free. Restore_free calculates a worst-case estimation on how much RAM is needed for the operation. When not enough RAM is available, the recv fails with ENOMEM.
Updates to very large sparse files lead to very large "free"-items and in turn to large allocation attempts (several GB).
I suggest changing restore_free to something along the line of

int
restore_free(struct restorearg *ra, objset_t *os,
                   struct drr_free *drrf)
{
        int err = 0;
        uint64_t offset = drrf->drr_offset;
        uint64_t length = drrf->drr_length;

        if (length != -1ULL && offset + length < offset)
                return (EINVAL);

        if (dmu_object_info(os, drrf->drr_object, NULL) != 0)
                return (EINVAL);

        while (length) {
                uint64_t l = length > 35184372088832ull ?
                             35184372088832ull : length;
                err = dmu_free_long_range(os, drrf->drr_object, offset, l);
                if (err)
                        break;
                offset += l;
                length -= l;
        }
        return (err);
}

i.e., processing the item in smaller chunks. In my short tests it works, but it is not properly tested.