Project

General

Profile

Bug #3190

zfs incremental: cannot receive incremental stream:destination ... has been modified since most recent snapshot

Added by Richard PALO about 8 years ago. Updated about 5 years ago.

Status:
Feedback
Priority:
Normal
Assignee:
-
Category:
zfs - Zettabyte File System
Start date:
2012-09-14
Due date:
% Done:

0%

Estimated time:
Difficulty:
Medium
Tags:
needs-triage
Gerrit CR:

Description

This was rather a frustrating discovery, running on oi_151a5:

bash-4.0# zfs list -rt all apool
NAME                                                   USED  AVAIL  REFER  MOUNTPOINT
apool                                                  111G   183G    97K  /apool
apool@2012-09-01                                        26K      -   100K  -
apool/richard                                          111G   183G  93,6G  /apool/richard
apool/richard@zfs-auto-snap_monthly-2012-07-06-06h37  2,17G      -  69,5G  -
apool/richard@zfs-auto-snap_weekly-2012-08-03-17h09   2,23G      -  74,6G  -
apool/richard@zfs-auto-snap_monthly-2012-08-06-07h11   132M      -  78,4G  -
apool/richard@zfs-auto-snap_weekly-2012-08-14-07h30    951M      -  79,2G  -
apool/richard@zfs-auto-snap_weekly-2012-08-21-07h30    351M      -  77,1G  -
apool/richard@zfs-auto-snap_daily-2012-08-28-07h19    81,3M      -  84,7G  -
apool/richard@zfs-auto-snap_weekly-2012-08-28-07h30   80,9M      -  84,7G  -
apool/richard@zfs-auto-snap_daily-2012-08-29-07h30     190M      -  84,7G  -
apool/richard@zfs-auto-snap_daily-2012-08-30-07h30    95,0M      -  92,4G  -
apool/richard@zfs-auto-snap_daily-2012-08-31-07h30    51,8M      -  92,5G  -
apool/richard@zfs-auto-snap_daily-2012-09-01-07h30    47,9M      -  92,5G  -
apool/richard@2012-09-01                               148M      -  93,1G  -
apool/richard@zfs-auto-snap_monthly-2012-09-11-19h25   116M      -  93,6G  -
apool/richard@zfs-auto-snap_daily-2012-09-13-08h13    84,5M      -  93,6G  -
bash-4.0# zfs list -rt all cpool/richard
NAME                                                   USED  AVAIL  REFER  MOUNTPOINT
cpool/richard                                          107G   112G  93,1G  /cpool/richard
cpool/richard@zfs-auto-snap_monthly-2012-07-06-06h37  3,41G      -  69,5G  -
cpool/richard@zfs-auto-snap_monthly-2012-08-06-07h11   146M      -  78,3G  -
cpool/richard@zfs-auto-snap_weekly-2012-08-14-07h30    951M      -  79,2G  -
cpool/richard@zfs-auto-snap_weekly-2012-08-21-07h30    351M      -  77,1G  -
cpool/richard@zfs-auto-snap_daily-2012-08-28-07h19    81,2M      -  84,7G  -
cpool/richard@zfs-auto-snap_weekly-2012-08-28-07h30   80,9M      -  84,7G  -
cpool/richard@zfs-auto-snap_daily-2012-08-29-07h30     190M      -  84,7G  -
cpool/richard@zfs-auto-snap_daily-2012-08-30-07h30    94,9M      -  92,4G  -
cpool/richard@zfs-auto-snap_daily-2012-08-31-07h30    51,7M      -  92,5G  -
cpool/richard@zfs-auto-snap_daily-2012-09-01-07h30    47,9M      -  92,5G  -
cpool/richard@2012-09-01                              1,04M      -  93,1G  -
cpool/richard@zfs-auto-snap_daily-2012-09-03-19h13     749K      -  93,1G  -
cpool/richard@zfs-auto-snap_weekly-2012-09-06-19h13       0      -  93,1G  -
bash-4.0# zfs send -i apool/richard@2012-09-01 apool/richard@zfs-auto-snap_daily-2012-09-13-08h13 | zfs receive cpool/richard
cannot receive incremental stream: most recent snapshot of cpool/richard does not
match incremental source
bash-4.0# zfs destroy cpool/richard@zfs-auto-snap_weekly-2012-09-06-19h13
bash-4.0# zfs destroy cpool/richard@zfs-auto-snap_daily-2012-09-03-19h13
bash-4.0# zfs list -rt all cpool/richard
NAME                                                   USED  AVAIL  REFER  MOUNTPOINT
cpool/richard                                          107G   112G  93,1G  /cpool/richard
cpool/richard@zfs-auto-snap_monthly-2012-07-06-06h37  3,41G      -  69,5G  -
cpool/richard@zfs-auto-snap_monthly-2012-08-06-07h11   146M      -  78,3G  -
cpool/richard@zfs-auto-snap_weekly-2012-08-14-07h30    951M      -  79,2G  -
cpool/richard@zfs-auto-snap_weekly-2012-08-21-07h30    351M      -  77,1G  -
cpool/richard@zfs-auto-snap_daily-2012-08-28-07h19    81,2M      -  84,7G  -
cpool/richard@zfs-auto-snap_weekly-2012-08-28-07h30   80,9M      -  84,7G  -
cpool/richard@zfs-auto-snap_daily-2012-08-29-07h30     190M      -  84,7G  -
cpool/richard@zfs-auto-snap_daily-2012-08-30-07h30    94,9M      -  92,4G  -
cpool/richard@zfs-auto-snap_daily-2012-08-31-07h30    51,7M      -  92,5G  -
cpool/richard@zfs-auto-snap_daily-2012-09-01-07h30    47,9M      -  92,5G  -
cpool/richard@2012-09-01                              1,04M      -  93,1G  -
bash-4.0# 
bash-4.0# zfs send -i apool/richard@2012-09-01 apool/richard@zfs-auto-snap_daily-2012-09-13-08h13 | zfs receive cpool/richard
cannot receive incremental stream: destination cpool/richard has been modified
since most recent snapshot
bash-4.0# 

As indicated, apool is the live volume and cpool is for all the backups, which gets mounted (import/export) on the same host.

Unfortunately, autosnapshot seems to have executed, but even when deleting all more recent snapshots on the backup disk to correspond to the beginning incremental snapshot, zfs spits up the error.

Is this intended behaviour?

History

#1

Updated by Richard PALO about 8 years ago

I did notice this oldish message:
news://news.gmane.org:
I've since noted that com.sun:auto-snapshot should be verified in the destination dataset.

It seems unintuitive that if an administrator destroys all snapshots after the reference dataset snapshot (as in this case) that the subsequent incremental send/receive should fail.
Would promoting the latest snapshot on the destination dataset after snapshot deletion help?

#2

Updated by Jon Tibble about 8 years ago

This line is your issue:

cpool/richard@2012-09-01 1,04M - 93,1G -

1,04M != 0 and therefore is modified on the destination exactly as the error message indicates. (But yes doing more snapshots on the destination is also unhelpful.)

See also zfs rollback - be careful you run this on the correct zpool though!

#3

Updated by Richard PALO about 8 years ago

  • Status changed from New to Feedback

Guess I needed to come across this case in point to better understand the mechanism... and duh!, rolling back is much more intuitive, just never really thought of that for a backup...

In other words, why would deleting the last couple of snapshots be different than rolling back (presuming a static backup dataset...) still scratching my head on that one, but oh well, kudos for the confirmation on the unambiguous way to achieve the intended snapshot state.
closing to feedback status...

#4

Updated by Richard PALO about 8 years ago

just 'discovered' zfs diff as well... Too bad it wasn't in the man pages.

https://www.illumos.org/issues/545

#5

Updated by Chip Schweiss about 5 years ago

There's more to this problem. Doing a 'zfs rollback' before an incremental send does not always fix the problem as it should.

I don't want to use -F on 'zfs receive' because I want all snapshot and folder destruction to be done manually on the receiving side. This way if someone accidentally does a 'zfs destroy' on the production side, the next replication cycle will not destroy it. In the case of my production storage this is 1 minute intervals. It also allows me to keep different automated snapshot scheduled on the production and DR copied.

Here's my operations:

zfs rollback -RF drpool/NIL02@.ozmt-replication_2015-08-05_15:26:49-0500
zfs send -P -v -R -I mirpool01/NIL02@.ozmt-replication_2015-08-05_15:26:49-0500 mirpool01/NIL02@.ozmt-replication_2015-08-06_16:05:17-0500 | zfs receive -v drpool/NIL02

It dies:

incremental     .ozmt-replication_2015-08-05_15:26:49-0500      mirpool01/NIL02@.ozmt-replication_2015-08-06_16:05:17-0500      0
incremental     .ozmt-replication_2015-08-05_15:26:49-0500      mirpool01/NIL02/CULVER@.ozmt-replication_2015-08-06_16:05:17-0500       0
incremental     .ozmt-replication_2015-08-05_15:26:49-0500      mirpool01/NIL02/CULVER/culver@weekday_2015-08-06_00:00-0500     9577253832
incremental     weekday_2015-08-06_00:00-0500   mirpool01/NIL02/CULVER/culver@.ozmt-replication_2015-08-06_16:05:17-0500        3815640
incremental     .ozmt-replication_2015-08-05_15:26:49-0500      mirpool01/NIL02/CVL@.ozmt-replication_2015-08-06_16:05:17-0500  0
incremental     .ozmt-replication_2015-08-05_15:26:49-0500      mirpool01/NIL02/CVL/cvlf@weekday_2015-08-06_00:00-0500  15497112
incremental     weekday_2015-08-06_00:00-0500   mirpool01/NIL02/CVL/cvlf@.ozmt-replication_2015-08-06_16:05:17-0500     17240104
incremental     .ozmt-replication_2015-08-05_15:26:49-0500      mirpool01/NIL02/CVL/cvl@weekday_2015-08-06_00:00-0500   1178666984
incremental     weekday_2015-08-06_00:00-0500   mirpool01/NIL02/CVL/cvl@.ozmt-replication_2015-08-06_16:05:17-0500      33644032
incremental     .ozmt-replication_2015-08-05_15:26:49-0500      mirpool01/NIL02/chendl@weekday_2015-08-06_00:00-0500    981256
incremental     weekday_2015-08-06_00:00-0500   mirpool01/NIL02/chendl@.ozmt-replication_2015-08-06_16:05:17-0500       1085376
incremental     .ozmt-replication_2015-08-05_15:26:49-0500      mirpool01/NIL02/CHENDL@.ozmt-replication_2015-08-06_16:05:17-0500       0
incremental     .ozmt-replication_2015-08-05_15:26:49-0500      mirpool01/NIL02/VLASSENKO@.ozmt-replication_2015-08-06_16:05:17-0500    0
incremental     .ozmt-replication_2015-08-05_15:26:49-0500      mirpool01/NIL02/VLASSENKO/vlassenko@weekday_2015-08-06_00:00-0500       8668305856
incremental     weekday_2015-08-06_00:00-0500   mirpool01/NIL02/VLASSENKO/vlassenko@.ozmt-replication_2015-08-06_16:05:17-0500  5883296
incremental     .ozmt-replication_2015-08-05_15:26:49-0500      mirpool01/NIL02/LAFOREST@.ozmt-replication_2015-08-06_16:05:17-0500     0
WARNING: could not send mirpool01/NIL02/SNIDERL@.ozmt-replication_2015-08-06_16:05:17-0500: does not exist
WARNING: could not send mirpool01/NIL02/SNIDERL/ccsn@.ozmt-replication_2015-08-06_16:05:17-0500: does not exist
incremental     .ozmt-replication_2015-08-05_15:26:49-0500      mirpool01/NIL02/ANCES@.ozmt-replication_2015-08-06_16:05:17-0500        0
incremental     .ozmt-replication_2015-08-05_15:26:49-0500      mirpool01/NIL02/ANCES/ances@weekday_2015-08-06_00:00-0500       27012952
incremental     weekday_2015-08-06_00:00-0500   mirpool01/NIL02/ANCES/ances@.ozmt-replication_2015-08-06_16:05:17-0500  5370048
incremental     .ozmt-replication_2015-08-05_15:26:49-0500      mirpool01/NIL02/TAI@.ozmt-replication_2015-08-06_16:05:17-0500  0
incremental     .ozmt-replication_2015-08-05_15:26:49-0500      mirpool01/NIL02/TAI/tai@weekday_2015-08-06_00:00-0500   8192
incremental     weekday_2015-08-06_00:00-0500   mirpool01/NIL02/TAI/tai@.ozmt-replication_2015-08-06_16:05:17-0500      0
incremental     .ozmt-replication_2015-08-05_15:26:49-0500      mirpool01/NIL02/BOTTERON@.ozmt-replication_2015-08-06_16:05:17-0500     0
incremental     .ozmt-replication_2015-08-05_15:26:49-0500      mirpool01/NIL02/BOTTERON/botteron@weekday_2015-08-06_00:00-0500 0
incremental     weekday_2015-08-06_00:00-0500   mirpool01/NIL02/BOTTERON/botteron@.ozmt-replication_2015-08-06_16:05:17-0500    0
incremental     .ozmt-replication_2015-08-05_15:26:49-0500      mirpool01/NIL02/BOTTERON/botteronf@weekday_2015-08-06_00:00-0500        6332544
incremental     weekday_2015-08-06_00:00-0500   mirpool01/NIL02/BOTTERON/botteronf@.ozmt-replication_2015-08-06_16:05:17-0500   7305096
size    19548402320
08:34:21        15989088        mirpool01/NIL02/CULVER/culver@weekday_2015-08-06_00:00-0500
08:34:22        31089304        mirpool01/NIL02/CULVER/culver@weekday_2015-08-06_00:00-0500
08:34:23        161248752       mirpool01/NIL02/CULVER/culver@weekday_2015-08-06_00:00-0500
08:34:24        269903320       mirpool01/NIL02/CULVER/culver@weekday_2015-08-06_00:00-0500
08:34:25        269903320       mirpool01/NIL02/CULVER/culver@weekday_2015-08-06_00:00-0500
08:34:26        269903320       mirpool01/NIL02/CULVER/culver@weekday_2015-08-06_00:00-0500
08:34:27        269903320       mirpool01/NIL02/CULVER/culver@weekday_2015-08-06_00:00-0500
08:34:28        269903320       mirpool01/NIL02/CULVER/culver@weekday_2015-08-06_00:00-0500
08:34:29        269903320       mirpool01/NIL02/CULVER/culver@weekday_2015-08-06_00:00-0500
08:34:30        269903320       mirpool01/NIL02/CULVER/culver@weekday_2015-08-06_00:00-0500
08:34:31        269903320       mirpool01/NIL02/CULVER/culver@weekday_2015-08-06_00:00-0500
08:34:32        269903320       mirpool01/NIL02/CULVER/culver@weekday_2015-08-06_00:00-0500
08:34:33        269903320       mirpool01/NIL02/CULVER/culver@weekday_2015-08-06_00:00-0500
08:34:34        269903320       mirpool01/NIL02/CULVER/culver@weekday_2015-08-06_00:00-0500
08:34:35        269903320       mirpool01/NIL02/CULVER/culver@weekday_2015-08-06_00:00-0500
08:34:36        269903320       mirpool01/NIL02/CULVER/culver@weekday_2015-08-06_00:00-0500
08:34:37        269903320       mirpool01/NIL02/CULVER/culver@weekday_2015-08-06_00:00-0500
08:34:38        269903320       mirpool01/NIL02/CULVER/culver@weekday_2015-08-06_00:00-0500
08:34:39        269903320       mirpool01/NIL02/CULVER/culver@weekday_2015-08-06_00:00-0500
cannot receive incremental stream: destination drpool/NIL02/CULVER has been modified
since most recent snapshot

The source file system that it is having trouble with:

# zfs list -t all -r mirpool01/NIL02/CULVER
NAME                                                                       USED  AVAIL  REFER  MOUNTPOINT
mirpool01/NIL02/CULVER                                                    6.48T   866G   205K  /NIL02/CULVER
mirpool01/NIL02/CULVER@.ozmt-replication_2015-08-05_15:26:49-0500             0      -   205K  -
mirpool01/NIL02/CULVER@.ozmt-replication_2015-08-06_16:05:17-0500             0      -   205K  -
mirpool01/NIL02/CULVER@.ozmt-replication_2015-08-12_07:55:54-0500             0      -   205K  -
mirpool01/NIL02/CULVER/culver                                             6.48T   866G  6.41T  /NIL02/CULVER/culver
mirpool01/NIL02/CULVER/culver@weekday_2015-04-23_00:00-0500               4.02G      -  5.34T  -
mirpool01/NIL02/CULVER/culver@weekday_2015-04-24_00:00-0500               28.0M      -  5.34T  -
mirpool01/NIL02/CULVER/culver@weekday_2015-04-27_00:00-0500               28.2M      -  5.34T  -
mirpool01/NIL02/CULVER/culver@weekday_2015-04-28_00:00-0500               28.2M      -  5.35T  -
mirpool01/NIL02/CULVER/culver@weekday_2015-04-29_00:00-0500               28.1M      -  5.35T  -
mirpool01/NIL02/CULVER/culver@weekday_2015-04-30_00:00-0500               27.9M      -  5.35T  -
mirpool01/NIL02/CULVER/culver@weekday_2015-05-01_00:00-0500               27.9M      -  5.35T  -
mirpool01/NIL02/CULVER/culver@weekday_2015-05-04_00:00-0500               28.9M      -  5.35T  -
mirpool01/NIL02/CULVER/culver@weekday_2015-05-05_00:00-0500               28.0M      -  5.35T  -
mirpool01/NIL02/CULVER/culver@weekday_2015-05-06_00:00-0500                158M      -  5.35T  -
mirpool01/NIL02/CULVER/culver@weekday_2015-05-18_00:00-0500               29.4M      -  5.41T  -
mirpool01/NIL02/CULVER/culver@weekday_2015-05-19_00:00-0500               28.7M      -  5.51T  -
mirpool01/NIL02/CULVER/culver@weekday_2015-05-20_00:00-0500                121M      -  5.54T  -
mirpool01/NIL02/CULVER/culver@weekday_2015-05-21_00:00-0500                   0      -  5.55T  -
mirpool01/NIL02/CULVER/culver@weekday_2015-05-22_00:00-0500                   0      -  5.55T  -
mirpool01/NIL02/CULVER/culver@weekday_2015-06-03_00:00-0500               55.3M      -  5.84T  -
mirpool01/NIL02/CULVER/culver@weekday_2015-06-04_00:00-0500                493M      -  5.90T  -
mirpool01/NIL02/CULVER/culver@weekday_2015-06-11_00:00-0500               3.18G      -  6.01T  -
mirpool01/NIL02/CULVER/culver@weekday_2015-06-12_00:00-0500               51.6M      -  6.03T  -
mirpool01/NIL02/CULVER/culver@weekday_2015-06-15_00:00-0500               28.0M      -  6.03T  -
mirpool01/NIL02/CULVER/culver@weekday_2015-06-16_00:00-0500                156M      -  6.12T  -
mirpool01/NIL02/CULVER/culver@weekday_2015-06-17_00:00-0500               29.3M      -  6.15T  -
mirpool01/NIL02/CULVER/culver@weekday_2015-06-18_00:00-0500               90.3M      -  6.19T  -
mirpool01/NIL02/CULVER/culver@weekday_2015-06-19_00:00-0500                655M      -  6.23T  -
mirpool01/NIL02/CULVER/culver@weekday_2015-07-28_00:00-0500                874M      -  6.36T  -
mirpool01/NIL02/CULVER/culver@.ozmt-replication_2015-08-05_15:26:49-0500   308M      -  6.37T  -
mirpool01/NIL02/CULVER/culver@weekday_2015-08-06_00:00-0500               17.7M      -  6.38T  -
mirpool01/NIL02/CULVER/culver@.ozmt-replication_2015-08-06_16:05:17-0500  17.4M      -  6.38T  -
mirpool01/NIL02/CULVER/culver@weekday_2015-08-07_00:00-0500               26.6M      -  6.38T  -
mirpool01/NIL02/CULVER/culver@weekday_2015-08-10_00:00-0500               29.4M      -  6.40T  -
mirpool01/NIL02/CULVER/culver@weekday_2015-08-11_00:00-0500               29.3M      -  6.40T  -
mirpool01/NIL02/CULVER/culver@weekday_2015-08-12_00:00-0500               5.25M      -  6.41T  -
mirpool01/NIL02/CULVER/culver@.ozmt-replication_2015-08-12_07:55:54-0500      0      -  6.41T  -

The receiving file system just after the rollback:

root@mir-dr-zfs01:/root/tools/utils# zfs rollback -Rf drpool/NIL02@.ozmt-replication_2015-08-05_15:26:49-0500
root@mir-dr-zfs01:/root/tools/utils# echo $?
0
root@mir-dr-zfs01:/root/tools/utils# zfs list -t all -r drpool/NIL02/CULVER
NAME                                                                    USED  AVAIL  REFER  MOUNTPOINT
drpool/NIL02/CULVER                                                    6.52T   818G   236K  /NIL02/CULVER
drpool/NIL02/CULVER@.ozmt-replication_2015-03-27_14:31:02-0500          158K      -   236K  -
drpool/NIL02/CULVER@.ozmt-replication_2015-08-05_15:26:49-0500          158K      -   236K  -
drpool/NIL02/CULVER/culver                                             6.52T   818G  6.42T  /NIL02/CULVER/culver
drpool/NIL02/CULVER/culver@.ozmt-replication_2015-03-27_14:31:02-0500  7.64G      -  5.07T  -
drpool/NIL02/CULVER/culver@weekday_2015-04-16_00:00-0500               20.0G      -  5.41T  -
drpool/NIL02/CULVER/culver@weekday_2015-04-17_00:00-0500               10.0G      -  5.40T  -
drpool/NIL02/CULVER/culver@weekday_2015-04-20_00:00-0500               31.0M      -  5.39T  -
drpool/NIL02/CULVER/culver@weekday_2015-04-21_00:00-0500                159M      -  5.39T  -
drpool/NIL02/CULVER/culver@weekday_2015-04-22_00:00-0500               30.8M      -  5.39T  -
drpool/NIL02/CULVER/culver@weekday_2015-04-23_00:00-0500               30.8M      -  5.39T  -
drpool/NIL02/CULVER/culver@weekday_2015-04-24_00:00-0500               30.7M      -  5.39T  -
drpool/NIL02/CULVER/culver@weekday_2015-04-27_00:00-0500               31.0M      -  5.39T  -
drpool/NIL02/CULVER/culver@weekday_2015-04-28_00:00-0500               31.0M      -  5.39T  -
drpool/NIL02/CULVER/culver@weekday_2015-04-29_00:00-0500               30.8M      -  5.40T  -
drpool/NIL02/CULVER/culver@weekday_2015-04-30_00:00-0500               30.6M      -  5.40T  -
drpool/NIL02/CULVER/culver@weekday_2015-05-01_00:00-0500               30.6M      -  5.40T  -
drpool/NIL02/CULVER/culver@weekday_2015-05-04_00:00-0500               31.7M      -  5.40T  -
drpool/NIL02/CULVER/culver@weekday_2015-05-05_00:00-0500               30.8M      -  5.40T  -
drpool/NIL02/CULVER/culver@weekday_2015-05-06_00:00-0500                161M      -  5.40T  -
drpool/NIL02/CULVER/culver@weekday_2015-05-18_00:00-0500               32.5M      -  5.46T  -
drpool/NIL02/CULVER/culver@weekday_2015-05-19_00:00-0500               31.5M      -  5.55T  -
drpool/NIL02/CULVER/culver@weekday_2015-05-20_00:00-0500                124M      -  5.58T  -
drpool/NIL02/CULVER/culver@weekday_2015-05-21_00:00-0500               19.7K      -  5.59T  -
drpool/NIL02/CULVER/culver@weekday_2015-05-22_00:00-0500               19.7K      -  5.59T  -
drpool/NIL02/CULVER/culver@weekday_2015-06-03_00:00-0500               58.2M      -  5.88T  -
drpool/NIL02/CULVER/culver@weekday_2015-06-04_00:00-0500                496M      -  5.94T  -
drpool/NIL02/CULVER/culver@weekday_2015-06-11_00:00-0500               3.18G      -  6.06T  -
drpool/NIL02/CULVER/culver@weekday_2015-06-12_00:00-0500               54.6M      -  6.08T  -
drpool/NIL02/CULVER/culver@weekday_2015-06-15_00:00-0500               30.8M      -  6.08T  -
drpool/NIL02/CULVER/culver@weekday_2015-06-16_00:00-0500                159M      -  6.16T  -
drpool/NIL02/CULVER/culver@weekday_2015-06-17_00:00-0500               32.1M      -  6.20T  -
drpool/NIL02/CULVER/culver@weekday_2015-06-18_00:00-0500               93.2M      -  6.23T  -
drpool/NIL02/CULVER/culver@weekday_2015-06-19_00:00-0500                731M      -  6.27T  -
drpool/NIL02/CULVER/culver@weekday_2015-07-28_00:00-0500                879M      -  6.41T  -
drpool/NIL02/CULVER/culver@.ozmt-replication_2015-08-05_15:26:49-0500      0      -  6.42T  -

Also available in: Atom PDF