Project

General

Profile

Feature #2605

want to resume interrupted zfs send

Added by Roy Sigurd Karlsbakk over 7 years ago. Updated about 4 years ago.

Status:
Closed
Priority:
Normal
Assignee:
Matthew Ahrens
Category:
zfs - Zettabyte File System
Start date:
2012-04-09
Due date:
% Done:

100%

Estimated time:
Difficulty:
Medium
Tags:
needs-triage

Description

With ZFS send/receive of large datasets, it would be rather nice to have an incremental or a resumable zfs send/receive system. in my current case, I have 50TiB uncompressed (which is how it's sent over the wire with zfs send/receive), so with the current hardware/compression/network bandwidth/etc, it'll take about two weeks to transfer it all. While time isn't a big issue in this, the chance of mbuffer/nc/whatever getting interrupted somehow is persistent. AFAICS the current code is somewhat incremental, in that it sends a snapshot at a time, and while this is good, it doesn't help much if the initial snapshot (after rotating them for a while, destroying old ones) contain 95% of the data.

So, would it be possible somehow to create a resumable snapshot transfer solution without rewriting zfs? Ideally, such a thing would allow for rebooting source and destination servers and not merely help stabilising the stream.

roy

History

#1

Updated by Eric Schrock over 7 years ago

  • Assignee set to Eric Schrock

Chris Siden at Delphix has a prototype of this that we'll get back to in the near future. He doesn't appear to have a bug account setup, so I'll assign this to myself.

#2

Updated by Michael Keller over 7 years ago

Isn't it possible to pause the pipe from zfs send by not reading anything from it and then later resume reading? The same for zfs receive?

If so a wrapper for the pipes will do (and how that exactly works is depended on the use case...).

The only case a zfs rewrite will be needed is if zfs send for the same snapshots will output every time different byte streams. In such a case a "resume" by skipping already captured output won't work if the system restarted while zfs send was paused.

#3

Updated by Christopher Siden over 7 years ago

  • Assignee changed from Eric Schrock to Christopher Siden
#4

Updated by Matthew Ahrens about 4 years ago

  • Subject changed from Partial/incremental ZFS send/receive to want to resume interrupted zfs send
  • Assignee changed from Christopher Siden to Matthew Ahrens
#5

Updated by Dan McDonald about 4 years ago

NOTE: Slides 20-26 in the slideshare URL pertain to this feature.

#6

Updated by Electric Monk about 4 years ago

  • Status changed from New to Closed
  • % Done changed from 0 to 100

git commit 9c3fd1216fa7fb02cfbc78a2518a686d54b48ab8

commit  9c3fd1216fa7fb02cfbc78a2518a686d54b48ab8
Author: Matthew Ahrens <mahrens@delphix.com>
Date:   2015-10-10T04:56:01.000Z

    2605 want to resume interrupted zfs send
    Reviewed by: George Wilson <george.wilson@delphix.com>
    Reviewed by: Paul Dagnelie <pcd@delphix.com>
    Reviewed by: Richard Elling <Richard.Elling@RichardElling.com>
    Reviewed by: Xin Li <delphij@freebsd.org>
    Reviewed by: Arne Jansen <sensille@gmx.net>
    Approved by: Dan McDonald <danmcd@omniti.com>

Also available in: Atom PDF