Feature #8727


Native data and metadata encryption for zfs

Added by Jorgen Lundman almost 6 years ago. Updated over 3 years ago.

Start date:
Due date:
% Done:


Estimated time:
Gerrit CR:
External Bug:


This is the ZOL PR by Tom Caputi, ported to IllumOS.

This change incorporates three major pieces:
The first change is a keystore that manages wrapping
and encryption keys for encrypted datasets. These
commands mostly involve manipulating the new
DSL Crypto Key ZAP Objects that live in the MOS. Each
encrypted dataset has its own DSL Crypto Key that is
protected with a user's key. This level of indirection
allows users to change their keys without re-encrypting
their entire datasets. The change implements the new
subcommands "zfs load-key", "zfs unload-key" and
"zfs change-key" which allow the user to manage their
encryption keys and settings. In addition, several new
flags and properties have been added to allow dataset
creation and to make mounting and unmounting more
The second piece of this patch provides the ability to
encrypt, decyrpt, and authenticate protected datasets.
Each object set maintains a Merkel tree of Message
Authentication Codes that protect the lower layers,
similarly to how checksums are maintained. This part
impacts the zio layer, which handles the actual
encryption and generation of MACs, as well as the ARC
and DMU, which need to be able to handle encrypted
buffers and protected data.
The last addition is the ability to do raw, encrypted
sends and receives. The idea here is to send raw
encrypted and compressed data and receive it exactly
as is on a backup system. This means that the dataset
on the receiving system is protected using the same
user key that is in use on the sending side. By doing
so, datasets can be efficiently backed up to an
untrusted system without fear of data being
Reviewed by: Matthew Ahrens <>
Reviewed-by: Brian Behlendorf <>
Reviewed-by: Jorgen Lundman <>
Signed-off-by: Tom Caputi <>

Send / Recv Fixes following b52563

This patch fixes several issues discovered after
the encryption patch was merged:
Fixed a bug where encrypted datasets could attempt
to receive embedded data records.
Fixed a bug where dirty records created by the recv
code wasn't properly setting the dr_raw flag.
Fixed a typo where a dmu_tx_commit() was changed to
Fixed a few error handling bugs unrelated to the
encryption patch in dmu_recv_stream()
Signed-off-by: Tom Caputi <>

Encryption patch follow-up

  • HKDF implementation moved to its own file and tests added to ensure
  • Ztest can now create and test encrypted datasets. This is currently
    disabled until issue ZOL #6526 is resolved, but otherwise functions as
  • Several small bug fixes discovered after enabling ztest to run on
    encrypted datasets.
  • Fixed coverity defects added by the encryption patch.
  • Updated man pages for encrypted send / receive behavior.
  • Fixed a bug where encrypted datasets could receive
  • Minor code cleanups / consolidation.

Disable crypto tests in ztest

  • Includes fix in dmu_free_long_object_impl

Unless permission is given to compile the crypto framework in userland
the crypto tests in ztest are disabled on IllumOS.

Related issues

Related to OpenIndiana Distribution - Feature #9117: ZFS disk encryption Resolved2018-02-15

Related to illumos gate - Bug #11278: Encrypted ZFS filesystems should be mounted at boot where possibleIn ProgressAndy Fiddaman

Has duplicate illumos gate - Bug #9952: Block size change during zfs receive drops spill blockDuplicateJerry Jelinek2018-11-03

Actions #1

Updated by Marcel Telka over 5 years ago

Actions #2

Updated by Jerry Jelinek over 4 years ago

  • Assignee set to Jerry Jelinek
Actions #3

Updated by Jerry Jelinek over 4 years ago

The following ZoL commits are included in this feature:

dd29864b0 Update raw send documentation
d93d4b1ac Revert "Fix issues with truncated files in raw sends"
5dbf8b4ed Fix issues with truncated files in raw sends
83472fabe Fix hierarchy misspellings
8b5814393 Incorrect maximum DVA value in DDE_GET_NDVAS()
52ce99dd6 Refcounted DSL Crypto Key Mappings
69830602d Raw receive fix and encrypted objset security fix
508c5527d Use 'printf %s' instead of 'echo n' for compatibility
eaed84054 Better user experience for errata 4
f00ab3f22 Detect and prevent mixed raw and non-raw sends
579ce7c5a Add bookmark v2 on-disk feature
369aa501d Fix handling of maxblkid for raw sends
1fff937a4c9 Check encrypted dataset + embedded recv earlier
d7e4b30a6 Add zfs_refcount_transfer_ownership_many()
52ce99dd6 Refcounted DSL Crypto Key Mappings
b7ddeaef3 Refactor arc_hdr_realloc_crypt
370bbf66a Fix coverity defects: CID 176037
ee45fbd89 ZFS send fails to dump objects larger than 128PiB
0d23f5e2e Fix hash_lock / keystore.sk_dk_lock lock inversion
cd32e5db8 Add ASSERT to debug encryption key mapping issues
b405837a6 Update the correct abd in l2arc_read_done()
e7504d7a1 Raw receive functions must not decrypt data
513168abd Make zvol update volsize operation synchronous.
be9a5c355 Add support for decryption faults in zinject
2c24b5b14 Fix issues found with zfs diff
b0ee5946a Fix issues with raw sends of spill blocks
edc1e713c Fix race in dnode_check_slots_free()
74df0c5e2 Correct swapped keylocation error messages
5c27ec108 Fixes for SNPRINTF_BLKPTR with encrypted BP's
1bf9a552b Make encrypted "zfs mount a" failures consistent
10adee27c Remove ASSERT in l2arc_apply_transforms()
a2c2ed1bd Decryption error handling improvements
4515b1d01 Encrypted dnode blocks should be prefetched raw
1a2342784 receive_spill does not byte swap spill contents
095495e00 Raw DRR_OBJECT records must write raw data
f8478fc2c Fix bounds check in zio_crypt_do_objset_hmacs
b0918402d Raw receive should change key atomically
4a385862b Prevent raw zfs recv F if dataset is unencrypted
b1d217338 Raw receives must compress metadnode blocks
478b3150d Add omitted set for os->os_next_write_raw
71a24c3c5 Handle compressed buffers in __dbuf_hold_impl()
1b66810ba Change os->os_next_write_raw to work per txg
047116ac7 Raw sends must be able to decrease nlevels
d53bd7f52 Fix recovery import (F) with encrypted pool
ae76f45cd Encryption Stability and On-Disk Format Fixes
62df1bc81 Fix encryption root hierarchy issue
21a932b83 Post-Encryption Followup
2637dda8f Fix for #6706
b135b9f11 Fix for #6703
440a3eb93 Fixes for #6639
4807c0bad Encryption patch follow-up
9b8407638 Send / Recv Fixes following b52563
b52563034 Native Encryption for ZFS on Linux

Actions #4

Updated by Jerry Jelinek over 4 years ago

Matt suggested we also include the following ZoL commit

c17bcf83da Enable raw writes to perform dedup with verification

Actions #5

Updated by Electric Monk over 4 years ago

  • Status changed from New to Closed
  • % Done changed from 0 to 100

git commit eb633035c80613ec93d62f90482837adaaf21a0a

commit  eb633035c80613ec93d62f90482837adaaf21a0a
Author: Tom Caputi <>
Date:   2019-06-25T19:40:06.000Z

    8727 Native data and metadata encryption for zfs
    Portions contributed by: Jorgen Lundman <>
    Portions contributed by: Jerry Jelinek <>
    Portions contributed by: Paul Zuchowski <>
    Portions contributed by: Tim Chase <>
    Portions contributed by: Matthew Ahrens <>
    Portions contributed by: ab-oe <>
    Portions contributed by: Brian Behlendorf <>
    Portions contributed by: loli10K <>
    Portions contributed by: Igor K <>
    Portions contributed by: Richard Laager <>
    Reviewed by: Jason Cohen <>
    Reviewed by: Allan Jude <>
    Reviewed by: George Melikov <>
    Reviewed by: Paul Dagnelie <>
    Reviewed by: RageLtMan <rageltman@sempervictus>
    Reviewed by: Matthew Thode <>
    Reviewed by: Giuseppe Di Natale <>
    Reviewed by: Kash Pande <>
    Reviewed by: Alek Pinchuk <>
    Reviewed by: Dan Kimmel <>
    Reviewed by: David Quigley <>
    Reviewed by: Jorgen Lundman <>
    Reviewed by: Matthew Ahrens <>
    Reviewed by: Brian Behlendorf <>
    Reviewed by: Toomas Soome <>
    Reviewed by: C Fraire <>
    Reviewed by: Jason King <>
    Reviewed by: Andy Stormont <>
    Approved by: Garrett D'Amore <>

Actions #6

Updated by Andy Fiddaman over 4 years ago

  • Related to Bug #11278: Encrypted ZFS filesystems should be mounted at boot where possible added
Actions #7

Updated by Jerry Jelinek over 3 years ago

  • Related to Bug #9952: Block size change during zfs receive drops spill block added
Actions #8

Updated by Jerry Jelinek over 3 years ago

The following additional openzfs commits, beyond the list above, are included with this commit:
caf9dd209 Fix send/recv lost spill block
c2c6eadf2 Fix issues with truncated files in raw sends
c17bcf83d Enable raw writes to perform dedup with verification

Actions #9

Updated by Joshua M. Clulow over 3 years ago

  • Has duplicate Bug #9952: Block size change during zfs receive drops spill block added
Actions #10

Updated by Joshua M. Clulow over 3 years ago

  • Related to deleted (Bug #9952: Block size change during zfs receive drops spill block)

Also available in: Atom PDF