Project

General

Profile

Actions
Copy link

Feature #2605

closed

want to resume interrupted zfs send

Added by Roy Sigurd Karlsbakk almost 11 years ago. Updated over 7 years ago.

Status:
Closed
Priority:
Normal
Assignee:
Matthew Ahrens
Category:
zfs - Zettabyte File System
Start date:
2012-04-09
Due date:
% Done:

100%

Estimated time:
Difficulty:
Medium
Tags:
needs-triage
Gerrit CR:
External Bug:

Description

With ZFS send/receive of large datasets, it would be rather nice to have an incremental or a resumable zfs send/receive system. in my current case, I have 50TiB uncompressed (which is how it's sent over the wire with zfs send/receive), so with the current hardware/compression/network bandwidth/etc, it'll take about two weeks to transfer it all. While time isn't a big issue in this, the chance of mbuffer/nc/whatever getting interrupted somehow is persistent. AFAICS the current code is somewhat incremental, in that it sends a snapshot at a time, and while this is good, it doesn't help much if the initial snapshot (after rotating them for a while, destroying old ones) contain 95% of the data.

So, would it be possible somehow to create a resumable snapshot transfer solution without rewriting zfs? Ideally, such a thing would allow for rebooting source and destination servers and not merely help stabilising the stream.

roy

Actions
Copy link
 #1

Updated by Eric Schrock almost 11 years ago

  • Assignee set to Eric Schrock

Chris Siden at Delphix has a prototype of this that we'll get back to in the near future. He doesn't appear to have a bug account setup, so I'll assign this to myself.

Actions
Copy link
 #2

Updated by Michael Keller over 10 years ago

Isn't it possible to pause the pipe from zfs send by not reading anything from it and then later resume reading? The same for zfs receive?

If so a wrapper for the pipes will do (and how that exactly works is depended on the use case...).

The only case a zfs rewrite will be needed is if zfs send for the same snapshots will output every time different byte streams. In such a case a "resume" by skipping already captured output won't work if the system restarted while zfs send was paused.

Actions
Copy link
 #3

Updated by Christopher Siden over 10 years ago

  • Assignee changed from Eric Schrock to Christopher Siden
Actions
Copy link
 #4

Updated by Matthew Ahrens over 7 years ago

  • Subject changed from Partial/incremental ZFS send/receive to want to resume interrupted zfs send
  • Assignee changed from Christopher Siden to Matthew Ahrens
Actions
Copy link
 #5

Updated by Dan McDonald over 7 years ago

NOTE: Slides 20-26 in the slideshare URL pertain to this feature.

Actions
Copy link
 #6

Updated by Electric Monk over 7 years ago

  • Status changed from New to Closed
  • % Done changed from 0 to 100

git commit 9c3fd1216fa7fb02cfbc78a2518a686d54b48ab8

commit  9c3fd1216fa7fb02cfbc78a2518a686d54b48ab8
Author: Matthew Ahrens <mahrens@delphix.com>
Date:   2015-10-10T04:56:01.000Z

    2605 want to resume interrupted zfs send
    Reviewed by: George Wilson <george.wilson@delphix.com>
    Reviewed by: Paul Dagnelie <pcd@delphix.com>
    Reviewed by: Richard Elling <Richard.Elling@RichardElling.com>
    Reviewed by: Xin Li <delphij@freebsd.org>
    Reviewed by: Arne Jansen <sensille@gmx.net>
    Approved by: Dan McDonald <danmcd@omniti.com>

Actions
Copy link

Also available in: Atom PDF