Feature #2605
want to resume interrupted zfs send
100%
Description
With ZFS send/receive of large datasets, it would be rather nice to have an incremental or a resumable zfs send/receive system. in my current case, I have 50TiB uncompressed (which is how it's sent over the wire with zfs send/receive), so with the current hardware/compression/network bandwidth/etc, it'll take about two weeks to transfer it all. While time isn't a big issue in this, the chance of mbuffer/nc/whatever getting interrupted somehow is persistent. AFAICS the current code is somewhat incremental, in that it sends a snapshot at a time, and while this is good, it doesn't help much if the initial snapshot (after rotating them for a while, destroying old ones) contain 95% of the data.
So, would it be possible somehow to create a resumable snapshot transfer solution without rewriting zfs? Ideally, such a thing would allow for rebooting source and destination servers and not merely help stabilising the stream.
roy
Updated by Eric Schrock almost 9 years ago
- Assignee set to Eric Schrock
Chris Siden at Delphix has a prototype of this that we'll get back to in the near future. He doesn't appear to have a bug account setup, so I'll assign this to myself.
Updated by Michael Keller almost 9 years ago
Isn't it possible to pause the pipe from zfs send by not reading anything from it and then later resume reading? The same for zfs receive?
If so a wrapper for the pipes will do (and how that exactly works is depended on the use case...).
The only case a zfs rewrite will be needed is if zfs send for the same snapshots will output every time different byte streams. In such a case a "resume" by skipping already captured output won't work if the system restarted while zfs send was paused.
Updated by Christopher Siden over 8 years ago
- Assignee changed from Eric Schrock to Christopher Siden
Updated by Matthew Ahrens over 5 years ago
- Subject changed from Partial/incremental ZFS send/receive to want to resume interrupted zfs send
- Assignee changed from Christopher Siden to Matthew Ahrens
See also slides http://www.slideshare.net/MatthewAhrens/openzfs-send-and-receive
and video https://www.youtube.com/watch?v=iY44jPMvxog
for more details.
Updated by Dan McDonald over 5 years ago
NOTE: Slides 20-26 in the slideshare URL pertain to this feature.
Updated by Electric Monk over 5 years ago
- Status changed from New to Closed
- % Done changed from 0 to 100
git commit 9c3fd1216fa7fb02cfbc78a2518a686d54b48ab8
commit 9c3fd1216fa7fb02cfbc78a2518a686d54b48ab8 Author: Matthew Ahrens <mahrens@delphix.com> Date: 2015-10-10T04:56:01.000Z 2605 want to resume interrupted zfs send Reviewed by: George Wilson <george.wilson@delphix.com> Reviewed by: Paul Dagnelie <pcd@delphix.com> Reviewed by: Richard Elling <Richard.Elling@RichardElling.com> Reviewed by: Xin Li <delphij@freebsd.org> Reviewed by: Arne Jansen <sensille@gmx.net> Approved by: Dan McDonald <danmcd@omniti.com>