illumos gate

This issue also seems to show up with snapshots that are at the refquota, rather than gratuitously exceeding it. On my OmniOS r151010 test VM, a reproduction case is:

1. zfs create rpool/foo
2. zfs set refquota=150M rpool/foo
3. yes test > /rpool/foo/fillerup
4. zfs snapshot rpool/foo@1
5. zfs snapshot rpool/foo@2
6. zfs send -R rpool/foo@2 | zfs recv rpool/bar

The most succinct way to see this bug is as follows, where the rpool/foo refquota is 150M:

bloody(~)[0]% zfs list -t all rpool/foo rpool/foo@1 rpool/foo@2
NAME          USED  AVAIL  REFER  MOUNTPOINT
rpool/foo     200M  50.0M   100M  /rpool/foo
rpool/foo@1   100M      -   200M  -
rpool/foo@2      0      -   100M  -
bloody(~)[0]% sudo zfs send -R rpool/foo@2 | sudo zfs recv rpool/bar
cannot receive new filesystem stream: destination rpool space quota exceeded
bloody(~)[1]% 

And the error only prints after a delay, suggesting the act of sending actually occurred. Confirmed this with a cross-pool send, same problem, longer delay because the other pool I used is spinning-rust (vs. an SSD rpool).

Here's an analysis.

A replicated stream dump has a header, containing among other things, and nvlist of properties for the filesystem. The problem is, those properties accompany EVERY SNAPSHOT, each one of which corresponds to a ZFS_IOC_RECV ioctl.

So in my above example, the intermediate snapshot exceeds the refquota, but it shouldn't be enforced, as refquotas only matter for the current filesystem, not its snapshots.

We need to set the properties in the header only on the LAST SNAPSHOT. This is indicated by "tosnap" in the main header.

Here is a backtrace of the failure from Linux:

 zfs  3035 [000]   186.746971: zfs:zfs_set__error: ffffffffa0bbaeac:2901:ffffffffa0b95df0(): error 0x55
        ffffffffa0abea93 dsl_dataset_check_quota (/lib/modules/3.12.21-gentoo-r1/extra/zfs/zfs.ko)
        ffffffffa0acd402 dsl_dir_tempreserve_impl (/lib/modules/3.12.21-gentoo-r1/extra/zfs/zfs.ko)
        ffffffffa0acef13 dsl_dir_tempreserve_space (/lib/modules/3.12.21-gentoo-r1/extra/zfs/zfs.ko)
        ffffffffa0ab0ec8 dmu_tx_assign (/lib/modules/3.12.21-gentoo-r1/extra/zfs/zfs.ko)
        ffffffffa0aa8255 restore_write (/lib/modules/3.12.21-gentoo-r1/extra/zfs/zfs.ko)
        ffffffffa0aab5c1 dmu_recv_stream (/lib/modules/3.12.21-gentoo-r1/extra/zfs/zfs.ko)
        ffffffffa0b4a1eb zfs_ioc_recv (/lib/modules/3.12.21-gentoo-r1/extra/zfs/zfs.ko)
        ffffffffa0b4c1ba zfsdev_ioctl (/lib/modules/3.12.21-gentoo-r1/extra/zfs/zfs.ko)
        ffffffff811909dd do_vfs_ioctl ([kernel.kallsyms])
        ffffffff81190c41 sys_ioctl ([kernel.kallsyms])
        ffffffff8156e2e9 system_call_fastpath ([kernel.kallsyms])
            7f8c676969c7 __GI___ioctl (/lib64/libc-2.19.so)
            7f8c67db3ae6 zfs_receive_one (/lib64/libzfs.so.2.0.0)
            7f8c67db4ebc zfs_receive_impl (/lib64/libzfs.so.2.0.0)
            7f8c67db22d4 zfs_receive_package (/lib64/libzfs.so.2.0.0)
            7f8c67db4f3a zfs_receive_impl (/lib64/libzfs.so.2.0.0)
            7f8c67db512f zfs_receive (/lib64/libzfs.so.2.0.0)
                  40dcea zfs_do_receive (/sbin/zfs)
                  4140d6 main (/sbin/zfs)
            7f8c675d7dc5 __libc_start_main (/lib64/libc-2.19.so)

From my talk with Dan last night, it seems that enforcing the quota on a receive stream does not make sense, so I think we can fix this by disabling the quota at the start of `dmu_recv_stream()` and restoring it at the end. I am inclined to try to workaround this by doing something like this:

diff --git a/module/zfs/dmu_send.c b/module/zfs/dmu_send.c
index b2d844e..2914f96 100644
--- a/module/zfs/dmu_send.c
+++ b/module/zfs/dmu_send.c
@@ -1856,6 +1856,7 @@ dmu_recv_stream(dmu_recv_cookie_t *drc, vnode_t *vp, offset_t *voffp,
        objset_t *os;
        zio_cksum_t pcksum;
        int featureflags;
+       uint64_t cached_dsquota;

        ra.byteswap = drc->drc_byteswap;
        ra.cksum = drc->drc_cksum;
@@ -1878,6 +1879,10 @@ dmu_recv_stream(dmu_recv_cookie_t *drc, vnode_t *vp, offset_t *voffp,

        featureflags = DMU_GET_FEATUREFLAGS(drc->drc_drrb->drr_versioninfo);

+       /* Temporarily disable quota restriction */
+       cached_dsquota = drc->drc_ds->ds_quota;
+       drc->drc_ds->ds_quota = 0;
+
        /* if this stream is dedup'ed, set up the avl tree for guid mapping */
        if (featureflags & DMU_BACKUP_FEATURE_DEDUP) {
                minor_t minor;
@@ -1999,6 +2004,7 @@ dmu_recv_stream(dmu_recv_cookie_t *drc, vnode_t *vp, offset_t *voffp,
        ASSERT(ra.err != 0);

 out:
+       drc->drc_ds->ds_quota = cached_dsquota;
        if ((featureflags & DMU_BACKUP_FEATURE_DEDUP) && (cleanup_fd != -1))
                zfs_onexit_fd_rele(cleanup_fd);

Unfortunately, this does not make Brian's test case work, but it does prevent me from seeing EDQUOT in SET_ERROR, which suggests to me there is one other place in the codebase that would need to change for this workaround to be effective. I cannot spare much more time to look at this right now, but I hope this is helpful to others looking into this.

I went through that entire trace-it-in-the-kernel bit already, and tried several zfs_ioc_recv() mods. That's not the answer.

Conceptually, the libzfs handling of a recursive receive stream needs to NOT send down the properties of a filesystem in a call to ZFS_IOC_RECV until the very last snapshot (the one that matches "tosnap" on the stream header) of the recursive stream.

You need to at least set some of the properties before doing the zfs_ioc_recv() -- those that affect the data being written, e.g. compression.