Project

General

Profile

Bug #4986

receiving replication stream fails if any snapshot exceeds refquota

Added by Lauri Tirkkonen about 5 years ago. Updated almost 4 years ago.

Status:
Closed
Priority:
Normal
Assignee:
-
Category:
zfs - Zettabyte File System
Start date:
2014-07-10
Due date:
% Done:

100%

Estimated time:
Difficulty:
Medium
Tags:
needs-triage

Description

  1. zfs create -o compress=off -o dedup=off rpool/foo
  2. mkfile 100M /rpool/foo/bar
  3. mkfile 100M /rpool/foo/baz
  4. zfs snapshot rpool/foo@1
  5. rm /rpool/foo/baz
  6. zfs set refquota=150M rpool/foo
  7. zfs list rpool/foo
    NAME USED AVAIL REFER MOUNTPOINT
    rpool/foo 200M 50.0M 100M /rpool/foo
  8. zfs snapshot rpool/foo@2
  9. zfs send -R rpool/foo@2 | zfs recv rpool/bar
    cannot receive new filesystem stream: destination rpool space quota exceeded

Related issues

Related to illumos gate - Bug #7998: zfs volumes sometimes grow on copyingNew2017-03-22

Actions

History

#1

Updated by Marcel Telka about 5 years ago

  • Category set to zfs - Zettabyte File System
#2

Updated by Brian De Wolf almost 5 years ago

This issue also seems to show up with snapshots that are at the refquota, rather than gratuitously exceeding it. On my OmniOS r151010 test VM, a reproduction case is:

1. zfs create rpool/foo
2. zfs set refquota=150M rpool/foo
3. yes test > /rpool/foo/fillerup
4. zfs snapshot rpool/foo@1
5. zfs snapshot rpool/foo@2
6. zfs send -R rpool/foo@2 | zfs recv rpool/bar

#3

Updated by Dan McDonald almost 4 years ago

The most succinct way to see this bug is as follows, where the rpool/foo refquota is 150M:

bloody(~)[0]% zfs list -t all rpool/foo rpool/foo@1 rpool/foo@2
NAME          USED  AVAIL  REFER  MOUNTPOINT
rpool/foo     200M  50.0M   100M  /rpool/foo
rpool/foo@1   100M      -   200M  -
rpool/foo@2      0      -   100M  -
bloody(~)[0]% sudo zfs send -R rpool/foo@2 | sudo zfs recv rpool/bar
cannot receive new filesystem stream: destination rpool space quota exceeded
bloody(~)[1]% 

And the error only prints after a delay, suggesting the act of sending actually occurred. Confirmed this with a cross-pool send, same problem, longer delay because the other pool I used is spinning-rust (vs. an SSD rpool).

#4

Updated by Dan McDonald almost 4 years ago

Here's an analysis.

A replicated stream dump has a header, containing among other things, and nvlist of properties for the filesystem. The problem is, those properties accompany EVERY SNAPSHOT, each one of which corresponds to a ZFS_IOC_RECV ioctl.

So in my above example, the intermediate snapshot exceeds the refquota, but it shouldn't be enforced, as refquotas only matter for the current filesystem, not its snapshots.

We need to set the properties in the header only on the LAST SNAPSHOT. This is indicated by "tosnap" in the main header.

#5

Updated by Richard Yao almost 4 years ago

Here is a backtrace of the failure from Linux:

 zfs  3035 [000]   186.746971: zfs:zfs_set__error: ffffffffa0bbaeac:2901:ffffffffa0b95df0(): error 0x55
        ffffffffa0abea93 dsl_dataset_check_quota (/lib/modules/3.12.21-gentoo-r1/extra/zfs/zfs.ko)
        ffffffffa0acd402 dsl_dir_tempreserve_impl (/lib/modules/3.12.21-gentoo-r1/extra/zfs/zfs.ko)
        ffffffffa0acef13 dsl_dir_tempreserve_space (/lib/modules/3.12.21-gentoo-r1/extra/zfs/zfs.ko)
        ffffffffa0ab0ec8 dmu_tx_assign (/lib/modules/3.12.21-gentoo-r1/extra/zfs/zfs.ko)
        ffffffffa0aa8255 restore_write (/lib/modules/3.12.21-gentoo-r1/extra/zfs/zfs.ko)
        ffffffffa0aab5c1 dmu_recv_stream (/lib/modules/3.12.21-gentoo-r1/extra/zfs/zfs.ko)
        ffffffffa0b4a1eb zfs_ioc_recv (/lib/modules/3.12.21-gentoo-r1/extra/zfs/zfs.ko)
        ffffffffa0b4c1ba zfsdev_ioctl (/lib/modules/3.12.21-gentoo-r1/extra/zfs/zfs.ko)
        ffffffff811909dd do_vfs_ioctl ([kernel.kallsyms])
        ffffffff81190c41 sys_ioctl ([kernel.kallsyms])
        ffffffff8156e2e9 system_call_fastpath ([kernel.kallsyms])
            7f8c676969c7 __GI___ioctl (/lib64/libc-2.19.so)
            7f8c67db3ae6 zfs_receive_one (/lib64/libzfs.so.2.0.0)
            7f8c67db4ebc zfs_receive_impl (/lib64/libzfs.so.2.0.0)
            7f8c67db22d4 zfs_receive_package (/lib64/libzfs.so.2.0.0)
            7f8c67db4f3a zfs_receive_impl (/lib64/libzfs.so.2.0.0)
            7f8c67db512f zfs_receive (/lib64/libzfs.so.2.0.0)
                  40dcea zfs_do_receive (/sbin/zfs)
                  4140d6 main (/sbin/zfs)
            7f8c675d7dc5 __libc_start_main (/lib64/libc-2.19.so)

From my talk with Dan last night, it seems that enforcing the quota on a receive stream does not make sense, so I think we can fix this by disabling the quota at the start of `dmu_recv_stream()` and restoring it at the end. I am inclined to try to workaround this by doing something like this:

diff --git a/module/zfs/dmu_send.c b/module/zfs/dmu_send.c
index b2d844e..2914f96 100644
--- a/module/zfs/dmu_send.c
+++ b/module/zfs/dmu_send.c
@@ -1856,6 +1856,7 @@ dmu_recv_stream(dmu_recv_cookie_t *drc, vnode_t *vp, offset_t *voffp,
        objset_t *os;
        zio_cksum_t pcksum;
        int featureflags;
+       uint64_t cached_dsquota;

        ra.byteswap = drc->drc_byteswap;
        ra.cksum = drc->drc_cksum;
@@ -1878,6 +1879,10 @@ dmu_recv_stream(dmu_recv_cookie_t *drc, vnode_t *vp, offset_t *voffp,

        featureflags = DMU_GET_FEATUREFLAGS(drc->drc_drrb->drr_versioninfo);

+       /* Temporarily disable quota restriction */
+       cached_dsquota = drc->drc_ds->ds_quota;
+       drc->drc_ds->ds_quota = 0;
+
        /* if this stream is dedup'ed, set up the avl tree for guid mapping */
        if (featureflags & DMU_BACKUP_FEATURE_DEDUP) {
                minor_t minor;
@@ -1999,6 +2004,7 @@ dmu_recv_stream(dmu_recv_cookie_t *drc, vnode_t *vp, offset_t *voffp,
        ASSERT(ra.err != 0);

 out:
+       drc->drc_ds->ds_quota = cached_dsquota;
        if ((featureflags & DMU_BACKUP_FEATURE_DEDUP) && (cleanup_fd != -1))
                zfs_onexit_fd_rele(cleanup_fd);

Unfortunately, this does not make Brian's test case work, but it does prevent me from seeing EDQUOT in SET_ERROR, which suggests to me there is one other place in the codebase that would need to change for this workaround to be effective. I cannot spare much more time to look at this right now, but I hope this is helpful to others looking into this.

#6

Updated by Dan McDonald almost 4 years ago

I went through that entire trace-it-in-the-kernel bit already, and tried several zfs_ioc_recv() mods. That's not the answer.

Conceptually, the libzfs handling of a recursive receive stream needs to NOT send down the properties of a filesystem in a call to ZFS_IOC_RECV until the very last snapshot (the one that matches "tosnap" on the stream header) of the recursive stream.

#7

Updated by Matthew Ahrens almost 4 years ago

You need to at least set some of the properties before doing the zfs_ioc_recv() -- those that affect the data being written, e.g. compression.

#8

Updated by Electric Monk almost 4 years ago

  • Status changed from New to Closed
  • % Done changed from 0 to 100

git commit 5878fad70d76d8711f6608c1f80b0447601261c6

commit  5878fad70d76d8711f6608c1f80b0447601261c6
Author: Dan McDonald <danmcd@omniti.com>
Date:   2015-12-08T01:36:38.000Z

    4986 receiving replication stream fails if any snapshot exceeds refquota
    Reviewed by: John Kennedy <john.kennedy@delphix.com>
    Reviewed by: Matthew Ahrens <mahrens@delphix.com>
    Approved by: Gordon Ross <gordon.ross@nexenta.com>

#9

Updated by Michael Mounteney over 2 years ago

  • Related to Bug #7998: zfs volumes sometimes grow on copying added

Also available in: Atom PDF