Feature #2605
closedwant to resume interrupted zfs send
100%
Description
With ZFS send/receive of large datasets, it would be rather nice to have an incremental or a resumable zfs send/receive system. in my current case, I have 50TiB uncompressed (which is how it's sent over the wire with zfs send/receive), so with the current hardware/compression/network bandwidth/etc, it'll take about two weeks to transfer it all. While time isn't a big issue in this, the chance of mbuffer/nc/whatever getting interrupted somehow is persistent. AFAICS the current code is somewhat incremental, in that it sends a snapshot at a time, and while this is good, it doesn't help much if the initial snapshot (after rotating them for a while, destroying old ones) contain 95% of the data.
So, would it be possible somehow to create a resumable snapshot transfer solution without rewriting zfs? Ideally, such a thing would allow for rebooting source and destination servers and not merely help stabilising the stream.
roy
Updated by Eric Schrock almost 11 years ago
- Assignee set to Eric Schrock
Chris Siden at Delphix has a prototype of this that we'll get back to in the near future. He doesn't appear to have a bug account setup, so I'll assign this to myself.
Updated by Michael Keller over 10 years ago
Isn't it possible to pause the pipe from zfs send by not reading anything from it and then later resume reading? The same for zfs receive?
If so a wrapper for the pipes will do (and how that exactly works is depended on the use case...).
The only case a zfs rewrite will be needed is if zfs send for the same snapshots will output every time different byte streams. In such a case a "resume" by skipping already captured output won't work if the system restarted while zfs send was paused.
Updated by Christopher Siden over 10 years ago
- Assignee changed from Eric Schrock to Christopher Siden
Updated by Matthew Ahrens over 7 years ago
- Subject changed from Partial/incremental ZFS send/receive to want to resume interrupted zfs send
- Assignee changed from Christopher Siden to Matthew Ahrens
See also slides http://www.slideshare.net/MatthewAhrens/openzfs-send-and-receive
and video https://www.youtube.com/watch?v=iY44jPMvxog
for more details.
Updated by Dan McDonald over 7 years ago
NOTE: Slides 20-26 in the slideshare URL pertain to this feature.
Updated by Electric Monk over 7 years ago
- Status changed from New to Closed
- % Done changed from 0 to 100
git commit 9c3fd1216fa7fb02cfbc78a2518a686d54b48ab8
commit 9c3fd1216fa7fb02cfbc78a2518a686d54b48ab8 Author: Matthew Ahrens <mahrens@delphix.com> Date: 2015-10-10T04:56:01.000Z 2605 want to resume interrupted zfs send Reviewed by: George Wilson <george.wilson@delphix.com> Reviewed by: Paul Dagnelie <pcd@delphix.com> Reviewed by: Richard Elling <Richard.Elling@RichardElling.com> Reviewed by: Xin Li <delphij@freebsd.org> Reviewed by: Arne Jansen <sensille@gmx.net> Approved by: Dan McDonald <danmcd@omniti.com>