Project

General

Profile

Bug #6435

zfs recursive send fails when descendant dataset deleted during send

Added by Brian De Wolf over 4 years ago.

Status:
New
Priority:
Normal
Assignee:
-
Category:
-
Start date:
2015-11-05
Due date:
% Done:

0%

Estimated time:
Difficulty:
Medium
Tags:
needs-triage
Gerrit CR:

Description

We use 'zfs send -R' to replicate a large tree of datasets. This tree sometimes has descendant datasets deleted, which tends to interfere with the zfs send operation. The zfs send might fail with one of two errors:

WARNING: could not send $descendant_dataset: does not exist

and/or
internal error: Unknown error

I was surprised by this because I expect zfs send to place holds on the snapshots it needs to complete the send operation to prevent the datasets from being deleted. I believe it does this correctly for the non-recursive operation, so it might just be a problem with recursive sends.

We worked around this issue by making our processes use locking to avoid running sends/deletions at the same time.

I was able to reproduce these problems on a fresh host using the following commands:

zfs create rpool/sendtest
for i in {1..250}; do zfs create rpool/sendtest/$i; done
zfs snapshot -r rpool/sendtest@1
zfs snapshot -r rpool/sendtest@2
zfs send -R rpool/sendtest@2 >/dev/null & zfs destroy -r rpool/sendtest/1 &>/dev/null
zfs send -R rpool/sendtest@2 >/dev/null & zfs destroy -r rpool/sendtest/2 &>/dev/null
zfs send -R rpool/sendtest@2 >/dev/null & zfs destroy -r rpool/sendtest/3 &>/dev/null
# and so on until the race condition shows the different errors

Here is output of these commands on OmniOS r151016:

root@r151016-test:/root# zfs create rpool/sendtest
root@r151016-test:/root# for i in {1..250}; do zfs create rpool/sendtest/$i; done
root@r151016-test:/root# zfs snapshot -r rpool/sendtest@1
root@r151016-test:/root# zfs snapshot -r rpool/sendtest@2
root@r151016-test:/root# zfs send -R rpool/sendtest@2 >/dev/null & zfs destroy -r rpool/sendtest/1 &>/dev/null
[1] 949
WARNING: could not send rpool/sendtest/1@2: does not exist
cannot open 'rpool/sendtest/1': dataset does not exist
root@r151016-test:/root# 
[1]+  Exit 1                  zfs send -R rpool/sendtest@2 > /dev/null
root@r151016-test:/root# zfs send -R rpool/sendtest@2 >/dev/null & zfs destroy -r rpool/sendtest/2 &>/dev/null
[1] 955
WARNING: could not send rpool/sendtest/2@2: does not exist
root@r151016-test:/root# cannot open 'rpool/sendtest/2': dataset does not exist

[1]+  Exit 1                  zfs send -R rpool/sendtest@2 > /dev/null
root@r151016-test:/root# 
root@r151016-test:/root# zfs send -R rpool/sendtest@2 >/dev/null & zfs destroy -r rpool/sendtest/3 &>/dev/null
[1] 963
cannot open 'rpool/sendtest/3': dataset does not exist
internal error: Unknown error
root@r151016-test:/root# 
[1]+  Abort                   (core dumped) zfs send -R rpool/sendtest@2 > /dev/null

Also available in: Atom PDF