want to resume interrupted zfs send
With ZFS send/receive of large datasets, it would be rather nice to have an incremental or a resumable zfs send/receive system. in my current case, I have 50TiB uncompressed (which is how it's sent over the wire with zfs send/receive), so with the current hardware/compression/network bandwidth/etc, it'll take about two weeks to transfer it all. While time isn't a big issue in this, the chance of mbuffer/nc/whatever getting interrupted somehow is persistent. AFAICS the current code is somewhat incremental, in that it sends a snapshot at a time, and while this is good, it doesn't help much if the initial snapshot (after rotating them for a while, destroying old ones) contain 95% of the data.
So, would it be possible somehow to create a resumable snapshot transfer solution without rewriting zfs? Ideally, such a thing would allow for rebooting source and destination servers and not merely help stabilising the stream.
Updated by Michael Keller over 10 years ago
Isn't it possible to pause the pipe from zfs send by not reading anything from it and then later resume reading? The same for zfs receive?
If so a wrapper for the pipes will do (and how that exactly works is depended on the use case...).
The only case a zfs rewrite will be needed is if zfs send for the same snapshots will output every time different byte streams. In such a case a "resume" by skipping already captured output won't work if the system restarted while zfs send was paused.
Updated by Matthew Ahrens over 7 years ago
- Subject changed from Partial/incremental ZFS send/receive to want to resume interrupted zfs send
- Assignee changed from Christopher Siden to Matthew Ahrens
See also slides http://www.slideshare.net/MatthewAhrens/openzfs-send-and-receive
and video https://www.youtube.com/watch?v=iY44jPMvxog
for more details.
Updated by Electric Monk over 7 years ago
- Status changed from New to Closed
- % Done changed from 0 to 100
commit 9c3fd1216fa7fb02cfbc78a2518a686d54b48ab8 Author: Matthew Ahrens <email@example.com> Date: 2015-10-10T04:56:01.000Z 2605 want to resume interrupted zfs send Reviewed by: George Wilson <firstname.lastname@example.org> Reviewed by: Paul Dagnelie <email@example.com> Reviewed by: Richard Elling <Richard.Elling@RichardElling.com> Reviewed by: Xin Li <firstname.lastname@example.org> Reviewed by: Arne Jansen <email@example.com> Approved by: Dan McDonald <firstname.lastname@example.org>