Bug #3190
zfs incremental: cannot receive incremental stream:destination ... has been modified since most recent snapshot
Status:
Feedback
Priority:
Normal
Assignee:
-
Category:
zfs - Zettabyte File System
Start date:
2012-09-14
Due date:
% Done:
0%
Estimated time:
Difficulty:
Medium
Tags:
needs-triage
Gerrit CR:
Description
This was rather a frustrating discovery, running on oi_151a5:
bash-4.0# zfs list -rt all apool NAME USED AVAIL REFER MOUNTPOINT apool 111G 183G 97K /apool apool@2012-09-01 26K - 100K - apool/richard 111G 183G 93,6G /apool/richard apool/richard@zfs-auto-snap_monthly-2012-07-06-06h37 2,17G - 69,5G - apool/richard@zfs-auto-snap_weekly-2012-08-03-17h09 2,23G - 74,6G - apool/richard@zfs-auto-snap_monthly-2012-08-06-07h11 132M - 78,4G - apool/richard@zfs-auto-snap_weekly-2012-08-14-07h30 951M - 79,2G - apool/richard@zfs-auto-snap_weekly-2012-08-21-07h30 351M - 77,1G - apool/richard@zfs-auto-snap_daily-2012-08-28-07h19 81,3M - 84,7G - apool/richard@zfs-auto-snap_weekly-2012-08-28-07h30 80,9M - 84,7G - apool/richard@zfs-auto-snap_daily-2012-08-29-07h30 190M - 84,7G - apool/richard@zfs-auto-snap_daily-2012-08-30-07h30 95,0M - 92,4G - apool/richard@zfs-auto-snap_daily-2012-08-31-07h30 51,8M - 92,5G - apool/richard@zfs-auto-snap_daily-2012-09-01-07h30 47,9M - 92,5G - apool/richard@2012-09-01 148M - 93,1G - apool/richard@zfs-auto-snap_monthly-2012-09-11-19h25 116M - 93,6G - apool/richard@zfs-auto-snap_daily-2012-09-13-08h13 84,5M - 93,6G - bash-4.0# zfs list -rt all cpool/richard NAME USED AVAIL REFER MOUNTPOINT cpool/richard 107G 112G 93,1G /cpool/richard cpool/richard@zfs-auto-snap_monthly-2012-07-06-06h37 3,41G - 69,5G - cpool/richard@zfs-auto-snap_monthly-2012-08-06-07h11 146M - 78,3G - cpool/richard@zfs-auto-snap_weekly-2012-08-14-07h30 951M - 79,2G - cpool/richard@zfs-auto-snap_weekly-2012-08-21-07h30 351M - 77,1G - cpool/richard@zfs-auto-snap_daily-2012-08-28-07h19 81,2M - 84,7G - cpool/richard@zfs-auto-snap_weekly-2012-08-28-07h30 80,9M - 84,7G - cpool/richard@zfs-auto-snap_daily-2012-08-29-07h30 190M - 84,7G - cpool/richard@zfs-auto-snap_daily-2012-08-30-07h30 94,9M - 92,4G - cpool/richard@zfs-auto-snap_daily-2012-08-31-07h30 51,7M - 92,5G - cpool/richard@zfs-auto-snap_daily-2012-09-01-07h30 47,9M - 92,5G - cpool/richard@2012-09-01 1,04M - 93,1G - cpool/richard@zfs-auto-snap_daily-2012-09-03-19h13 749K - 93,1G - cpool/richard@zfs-auto-snap_weekly-2012-09-06-19h13 0 - 93,1G - bash-4.0# zfs send -i apool/richard@2012-09-01 apool/richard@zfs-auto-snap_daily-2012-09-13-08h13 | zfs receive cpool/richard cannot receive incremental stream: most recent snapshot of cpool/richard does not match incremental source bash-4.0# zfs destroy cpool/richard@zfs-auto-snap_weekly-2012-09-06-19h13 bash-4.0# zfs destroy cpool/richard@zfs-auto-snap_daily-2012-09-03-19h13 bash-4.0# zfs list -rt all cpool/richard NAME USED AVAIL REFER MOUNTPOINT cpool/richard 107G 112G 93,1G /cpool/richard cpool/richard@zfs-auto-snap_monthly-2012-07-06-06h37 3,41G - 69,5G - cpool/richard@zfs-auto-snap_monthly-2012-08-06-07h11 146M - 78,3G - cpool/richard@zfs-auto-snap_weekly-2012-08-14-07h30 951M - 79,2G - cpool/richard@zfs-auto-snap_weekly-2012-08-21-07h30 351M - 77,1G - cpool/richard@zfs-auto-snap_daily-2012-08-28-07h19 81,2M - 84,7G - cpool/richard@zfs-auto-snap_weekly-2012-08-28-07h30 80,9M - 84,7G - cpool/richard@zfs-auto-snap_daily-2012-08-29-07h30 190M - 84,7G - cpool/richard@zfs-auto-snap_daily-2012-08-30-07h30 94,9M - 92,4G - cpool/richard@zfs-auto-snap_daily-2012-08-31-07h30 51,7M - 92,5G - cpool/richard@zfs-auto-snap_daily-2012-09-01-07h30 47,9M - 92,5G - cpool/richard@2012-09-01 1,04M - 93,1G - bash-4.0# bash-4.0# zfs send -i apool/richard@2012-09-01 apool/richard@zfs-auto-snap_daily-2012-09-13-08h13 | zfs receive cpool/richard cannot receive incremental stream: destination cpool/richard has been modified since most recent snapshot bash-4.0#
As indicated, apool is the live volume and cpool is for all the backups, which gets mounted (import/export) on the same host.
Unfortunately, autosnapshot seems to have executed, but even when deleting all more recent snapshots on the backup disk to correspond to the beginning incremental snapshot, zfs spits up the error.
Is this intended behaviour?
History
Updated by Richard PALO about 8 years ago
Updated by I did notice this oldish message:
news://news.gmane.org:119/F536C565-797B-41B6-BED4-809EB6BC5760@gmail.com
I've since noted that com.sun:auto-snapshot should be verified in the destination dataset.
It seems unintuitive that if an administrator destroys all snapshots after the reference dataset snapshot (as in this case) that the subsequent incremental send/receive should fail.
Would promoting the latest snapshot on the destination dataset after snapshot deletion help?
Updated by Jon Tibble about 8 years ago
Updated by This line is your issue:
cpool/richard@2012-09-01 1,04M - 93,1G -
1,04M != 0 and therefore is modified on the destination exactly as the error message indicates. (But yes doing more snapshots on the destination is also unhelpful.)
See also zfs rollback - be careful you run this on the correct zpool though!
Updated by Richard PALO about 8 years ago
- Status changed from New to Feedback
Guess I needed to come across this case in point to better understand the mechanism... and duh!, rolling back is much more intuitive, just never really thought of that for a backup...
In other words, why would deleting the last couple of snapshots be different than rolling back (presuming a static backup dataset...) still scratching my head on that one, but oh well, kudos for the confirmation on the unambiguous way to achieve the intended snapshot state.
closing to feedback status...
Updated by Richard PALO about 8 years ago
just 'discovered' zfs diff as well... Too bad it wasn't in the man pages.
Updated by Chip Schweiss about 5 years ago
Updated by There's more to this problem. Doing a 'zfs rollback' before an incremental send does not always fix the problem as it should.
I don't want to use -F on 'zfs receive' because I want all snapshot and folder destruction to be done manually on the receiving side. This way if someone accidentally does a 'zfs destroy' on the production side, the next replication cycle will not destroy it. In the case of my production storage this is 1 minute intervals. It also allows me to keep different automated snapshot scheduled on the production and DR copied.
Here's my operations:
zfs rollback -RF drpool/NIL02@.ozmt-replication_2015-08-05_15:26:49-0500 zfs send -P -v -R -I mirpool01/NIL02@.ozmt-replication_2015-08-05_15:26:49-0500 mirpool01/NIL02@.ozmt-replication_2015-08-06_16:05:17-0500 | zfs receive -v drpool/NIL02
It dies:
incremental .ozmt-replication_2015-08-05_15:26:49-0500 mirpool01/NIL02@.ozmt-replication_2015-08-06_16:05:17-0500 0 incremental .ozmt-replication_2015-08-05_15:26:49-0500 mirpool01/NIL02/CULVER@.ozmt-replication_2015-08-06_16:05:17-0500 0 incremental .ozmt-replication_2015-08-05_15:26:49-0500 mirpool01/NIL02/CULVER/culver@weekday_2015-08-06_00:00-0500 9577253832 incremental weekday_2015-08-06_00:00-0500 mirpool01/NIL02/CULVER/culver@.ozmt-replication_2015-08-06_16:05:17-0500 3815640 incremental .ozmt-replication_2015-08-05_15:26:49-0500 mirpool01/NIL02/CVL@.ozmt-replication_2015-08-06_16:05:17-0500 0 incremental .ozmt-replication_2015-08-05_15:26:49-0500 mirpool01/NIL02/CVL/cvlf@weekday_2015-08-06_00:00-0500 15497112 incremental weekday_2015-08-06_00:00-0500 mirpool01/NIL02/CVL/cvlf@.ozmt-replication_2015-08-06_16:05:17-0500 17240104 incremental .ozmt-replication_2015-08-05_15:26:49-0500 mirpool01/NIL02/CVL/cvl@weekday_2015-08-06_00:00-0500 1178666984 incremental weekday_2015-08-06_00:00-0500 mirpool01/NIL02/CVL/cvl@.ozmt-replication_2015-08-06_16:05:17-0500 33644032 incremental .ozmt-replication_2015-08-05_15:26:49-0500 mirpool01/NIL02/chendl@weekday_2015-08-06_00:00-0500 981256 incremental weekday_2015-08-06_00:00-0500 mirpool01/NIL02/chendl@.ozmt-replication_2015-08-06_16:05:17-0500 1085376 incremental .ozmt-replication_2015-08-05_15:26:49-0500 mirpool01/NIL02/CHENDL@.ozmt-replication_2015-08-06_16:05:17-0500 0 incremental .ozmt-replication_2015-08-05_15:26:49-0500 mirpool01/NIL02/VLASSENKO@.ozmt-replication_2015-08-06_16:05:17-0500 0 incremental .ozmt-replication_2015-08-05_15:26:49-0500 mirpool01/NIL02/VLASSENKO/vlassenko@weekday_2015-08-06_00:00-0500 8668305856 incremental weekday_2015-08-06_00:00-0500 mirpool01/NIL02/VLASSENKO/vlassenko@.ozmt-replication_2015-08-06_16:05:17-0500 5883296 incremental .ozmt-replication_2015-08-05_15:26:49-0500 mirpool01/NIL02/LAFOREST@.ozmt-replication_2015-08-06_16:05:17-0500 0 WARNING: could not send mirpool01/NIL02/SNIDERL@.ozmt-replication_2015-08-06_16:05:17-0500: does not exist WARNING: could not send mirpool01/NIL02/SNIDERL/ccsn@.ozmt-replication_2015-08-06_16:05:17-0500: does not exist incremental .ozmt-replication_2015-08-05_15:26:49-0500 mirpool01/NIL02/ANCES@.ozmt-replication_2015-08-06_16:05:17-0500 0 incremental .ozmt-replication_2015-08-05_15:26:49-0500 mirpool01/NIL02/ANCES/ances@weekday_2015-08-06_00:00-0500 27012952 incremental weekday_2015-08-06_00:00-0500 mirpool01/NIL02/ANCES/ances@.ozmt-replication_2015-08-06_16:05:17-0500 5370048 incremental .ozmt-replication_2015-08-05_15:26:49-0500 mirpool01/NIL02/TAI@.ozmt-replication_2015-08-06_16:05:17-0500 0 incremental .ozmt-replication_2015-08-05_15:26:49-0500 mirpool01/NIL02/TAI/tai@weekday_2015-08-06_00:00-0500 8192 incremental weekday_2015-08-06_00:00-0500 mirpool01/NIL02/TAI/tai@.ozmt-replication_2015-08-06_16:05:17-0500 0 incremental .ozmt-replication_2015-08-05_15:26:49-0500 mirpool01/NIL02/BOTTERON@.ozmt-replication_2015-08-06_16:05:17-0500 0 incremental .ozmt-replication_2015-08-05_15:26:49-0500 mirpool01/NIL02/BOTTERON/botteron@weekday_2015-08-06_00:00-0500 0 incremental weekday_2015-08-06_00:00-0500 mirpool01/NIL02/BOTTERON/botteron@.ozmt-replication_2015-08-06_16:05:17-0500 0 incremental .ozmt-replication_2015-08-05_15:26:49-0500 mirpool01/NIL02/BOTTERON/botteronf@weekday_2015-08-06_00:00-0500 6332544 incremental weekday_2015-08-06_00:00-0500 mirpool01/NIL02/BOTTERON/botteronf@.ozmt-replication_2015-08-06_16:05:17-0500 7305096 size 19548402320 08:34:21 15989088 mirpool01/NIL02/CULVER/culver@weekday_2015-08-06_00:00-0500 08:34:22 31089304 mirpool01/NIL02/CULVER/culver@weekday_2015-08-06_00:00-0500 08:34:23 161248752 mirpool01/NIL02/CULVER/culver@weekday_2015-08-06_00:00-0500 08:34:24 269903320 mirpool01/NIL02/CULVER/culver@weekday_2015-08-06_00:00-0500 08:34:25 269903320 mirpool01/NIL02/CULVER/culver@weekday_2015-08-06_00:00-0500 08:34:26 269903320 mirpool01/NIL02/CULVER/culver@weekday_2015-08-06_00:00-0500 08:34:27 269903320 mirpool01/NIL02/CULVER/culver@weekday_2015-08-06_00:00-0500 08:34:28 269903320 mirpool01/NIL02/CULVER/culver@weekday_2015-08-06_00:00-0500 08:34:29 269903320 mirpool01/NIL02/CULVER/culver@weekday_2015-08-06_00:00-0500 08:34:30 269903320 mirpool01/NIL02/CULVER/culver@weekday_2015-08-06_00:00-0500 08:34:31 269903320 mirpool01/NIL02/CULVER/culver@weekday_2015-08-06_00:00-0500 08:34:32 269903320 mirpool01/NIL02/CULVER/culver@weekday_2015-08-06_00:00-0500 08:34:33 269903320 mirpool01/NIL02/CULVER/culver@weekday_2015-08-06_00:00-0500 08:34:34 269903320 mirpool01/NIL02/CULVER/culver@weekday_2015-08-06_00:00-0500 08:34:35 269903320 mirpool01/NIL02/CULVER/culver@weekday_2015-08-06_00:00-0500 08:34:36 269903320 mirpool01/NIL02/CULVER/culver@weekday_2015-08-06_00:00-0500 08:34:37 269903320 mirpool01/NIL02/CULVER/culver@weekday_2015-08-06_00:00-0500 08:34:38 269903320 mirpool01/NIL02/CULVER/culver@weekday_2015-08-06_00:00-0500 08:34:39 269903320 mirpool01/NIL02/CULVER/culver@weekday_2015-08-06_00:00-0500 cannot receive incremental stream: destination drpool/NIL02/CULVER has been modified since most recent snapshot
The source file system that it is having trouble with:
# zfs list -t all -r mirpool01/NIL02/CULVER NAME USED AVAIL REFER MOUNTPOINT mirpool01/NIL02/CULVER 6.48T 866G 205K /NIL02/CULVER mirpool01/NIL02/CULVER@.ozmt-replication_2015-08-05_15:26:49-0500 0 - 205K - mirpool01/NIL02/CULVER@.ozmt-replication_2015-08-06_16:05:17-0500 0 - 205K - mirpool01/NIL02/CULVER@.ozmt-replication_2015-08-12_07:55:54-0500 0 - 205K - mirpool01/NIL02/CULVER/culver 6.48T 866G 6.41T /NIL02/CULVER/culver mirpool01/NIL02/CULVER/culver@weekday_2015-04-23_00:00-0500 4.02G - 5.34T - mirpool01/NIL02/CULVER/culver@weekday_2015-04-24_00:00-0500 28.0M - 5.34T - mirpool01/NIL02/CULVER/culver@weekday_2015-04-27_00:00-0500 28.2M - 5.34T - mirpool01/NIL02/CULVER/culver@weekday_2015-04-28_00:00-0500 28.2M - 5.35T - mirpool01/NIL02/CULVER/culver@weekday_2015-04-29_00:00-0500 28.1M - 5.35T - mirpool01/NIL02/CULVER/culver@weekday_2015-04-30_00:00-0500 27.9M - 5.35T - mirpool01/NIL02/CULVER/culver@weekday_2015-05-01_00:00-0500 27.9M - 5.35T - mirpool01/NIL02/CULVER/culver@weekday_2015-05-04_00:00-0500 28.9M - 5.35T - mirpool01/NIL02/CULVER/culver@weekday_2015-05-05_00:00-0500 28.0M - 5.35T - mirpool01/NIL02/CULVER/culver@weekday_2015-05-06_00:00-0500 158M - 5.35T - mirpool01/NIL02/CULVER/culver@weekday_2015-05-18_00:00-0500 29.4M - 5.41T - mirpool01/NIL02/CULVER/culver@weekday_2015-05-19_00:00-0500 28.7M - 5.51T - mirpool01/NIL02/CULVER/culver@weekday_2015-05-20_00:00-0500 121M - 5.54T - mirpool01/NIL02/CULVER/culver@weekday_2015-05-21_00:00-0500 0 - 5.55T - mirpool01/NIL02/CULVER/culver@weekday_2015-05-22_00:00-0500 0 - 5.55T - mirpool01/NIL02/CULVER/culver@weekday_2015-06-03_00:00-0500 55.3M - 5.84T - mirpool01/NIL02/CULVER/culver@weekday_2015-06-04_00:00-0500 493M - 5.90T - mirpool01/NIL02/CULVER/culver@weekday_2015-06-11_00:00-0500 3.18G - 6.01T - mirpool01/NIL02/CULVER/culver@weekday_2015-06-12_00:00-0500 51.6M - 6.03T - mirpool01/NIL02/CULVER/culver@weekday_2015-06-15_00:00-0500 28.0M - 6.03T - mirpool01/NIL02/CULVER/culver@weekday_2015-06-16_00:00-0500 156M - 6.12T - mirpool01/NIL02/CULVER/culver@weekday_2015-06-17_00:00-0500 29.3M - 6.15T - mirpool01/NIL02/CULVER/culver@weekday_2015-06-18_00:00-0500 90.3M - 6.19T - mirpool01/NIL02/CULVER/culver@weekday_2015-06-19_00:00-0500 655M - 6.23T - mirpool01/NIL02/CULVER/culver@weekday_2015-07-28_00:00-0500 874M - 6.36T - mirpool01/NIL02/CULVER/culver@.ozmt-replication_2015-08-05_15:26:49-0500 308M - 6.37T - mirpool01/NIL02/CULVER/culver@weekday_2015-08-06_00:00-0500 17.7M - 6.38T - mirpool01/NIL02/CULVER/culver@.ozmt-replication_2015-08-06_16:05:17-0500 17.4M - 6.38T - mirpool01/NIL02/CULVER/culver@weekday_2015-08-07_00:00-0500 26.6M - 6.38T - mirpool01/NIL02/CULVER/culver@weekday_2015-08-10_00:00-0500 29.4M - 6.40T - mirpool01/NIL02/CULVER/culver@weekday_2015-08-11_00:00-0500 29.3M - 6.40T - mirpool01/NIL02/CULVER/culver@weekday_2015-08-12_00:00-0500 5.25M - 6.41T - mirpool01/NIL02/CULVER/culver@.ozmt-replication_2015-08-12_07:55:54-0500 0 - 6.41T -
The receiving file system just after the rollback:
root@mir-dr-zfs01:/root/tools/utils# zfs rollback -Rf drpool/NIL02@.ozmt-replication_2015-08-05_15:26:49-0500 root@mir-dr-zfs01:/root/tools/utils# echo $? 0 root@mir-dr-zfs01:/root/tools/utils# zfs list -t all -r drpool/NIL02/CULVER NAME USED AVAIL REFER MOUNTPOINT drpool/NIL02/CULVER 6.52T 818G 236K /NIL02/CULVER drpool/NIL02/CULVER@.ozmt-replication_2015-03-27_14:31:02-0500 158K - 236K - drpool/NIL02/CULVER@.ozmt-replication_2015-08-05_15:26:49-0500 158K - 236K - drpool/NIL02/CULVER/culver 6.52T 818G 6.42T /NIL02/CULVER/culver drpool/NIL02/CULVER/culver@.ozmt-replication_2015-03-27_14:31:02-0500 7.64G - 5.07T - drpool/NIL02/CULVER/culver@weekday_2015-04-16_00:00-0500 20.0G - 5.41T - drpool/NIL02/CULVER/culver@weekday_2015-04-17_00:00-0500 10.0G - 5.40T - drpool/NIL02/CULVER/culver@weekday_2015-04-20_00:00-0500 31.0M - 5.39T - drpool/NIL02/CULVER/culver@weekday_2015-04-21_00:00-0500 159M - 5.39T - drpool/NIL02/CULVER/culver@weekday_2015-04-22_00:00-0500 30.8M - 5.39T - drpool/NIL02/CULVER/culver@weekday_2015-04-23_00:00-0500 30.8M - 5.39T - drpool/NIL02/CULVER/culver@weekday_2015-04-24_00:00-0500 30.7M - 5.39T - drpool/NIL02/CULVER/culver@weekday_2015-04-27_00:00-0500 31.0M - 5.39T - drpool/NIL02/CULVER/culver@weekday_2015-04-28_00:00-0500 31.0M - 5.39T - drpool/NIL02/CULVER/culver@weekday_2015-04-29_00:00-0500 30.8M - 5.40T - drpool/NIL02/CULVER/culver@weekday_2015-04-30_00:00-0500 30.6M - 5.40T - drpool/NIL02/CULVER/culver@weekday_2015-05-01_00:00-0500 30.6M - 5.40T - drpool/NIL02/CULVER/culver@weekday_2015-05-04_00:00-0500 31.7M - 5.40T - drpool/NIL02/CULVER/culver@weekday_2015-05-05_00:00-0500 30.8M - 5.40T - drpool/NIL02/CULVER/culver@weekday_2015-05-06_00:00-0500 161M - 5.40T - drpool/NIL02/CULVER/culver@weekday_2015-05-18_00:00-0500 32.5M - 5.46T - drpool/NIL02/CULVER/culver@weekday_2015-05-19_00:00-0500 31.5M - 5.55T - drpool/NIL02/CULVER/culver@weekday_2015-05-20_00:00-0500 124M - 5.58T - drpool/NIL02/CULVER/culver@weekday_2015-05-21_00:00-0500 19.7K - 5.59T - drpool/NIL02/CULVER/culver@weekday_2015-05-22_00:00-0500 19.7K - 5.59T - drpool/NIL02/CULVER/culver@weekday_2015-06-03_00:00-0500 58.2M - 5.88T - drpool/NIL02/CULVER/culver@weekday_2015-06-04_00:00-0500 496M - 5.94T - drpool/NIL02/CULVER/culver@weekday_2015-06-11_00:00-0500 3.18G - 6.06T - drpool/NIL02/CULVER/culver@weekday_2015-06-12_00:00-0500 54.6M - 6.08T - drpool/NIL02/CULVER/culver@weekday_2015-06-15_00:00-0500 30.8M - 6.08T - drpool/NIL02/CULVER/culver@weekday_2015-06-16_00:00-0500 159M - 6.16T - drpool/NIL02/CULVER/culver@weekday_2015-06-17_00:00-0500 32.1M - 6.20T - drpool/NIL02/CULVER/culver@weekday_2015-06-18_00:00-0500 93.2M - 6.23T - drpool/NIL02/CULVER/culver@weekday_2015-06-19_00:00-0500 731M - 6.27T - drpool/NIL02/CULVER/culver@weekday_2015-07-28_00:00-0500 879M - 6.41T - drpool/NIL02/CULVER/culver@.ozmt-replication_2015-08-05_15:26:49-0500 0 - 6.42T -