Feature #4896
open
Performance improvements for KCF AES modes
90%
Description
The KCF has numerous serious performance deficiencies under that severely impede its usability:
1) AES-NI acceleration code uses the CLTS instruction every 16 bytes of input, which causes VM exits when virtualized, severely impacting throughput
2) ECB, CTR, CBC and GCM mode implementations give very poor performance (lots of copying, no potential for pipelining)
3) Lack of assembly-optimized pipelineable versions of high-performance cipher combos (AES/CTR, AES/CBC)
4) Buffering of all ciphertext in the kernel when GCM decryption is performed.
This patch also includes work to improve the performance of the GHASH function in GCM ported over from the OpenSSL project.
More details can be found at: http://zfs-create.blogspot.com/2014/05/optimizing-illumos-kernel-crypto.html
Updated by Guenther Alka over 3 years ago
Is this still relevant?
Is Illumos performance with encrypted filesystems and aes-ni comparable to other platforms?
Updated by Trevor Talbot 7 months ago
Guenther Alka wrote in #note-1:
Is this still relevant?
Is Illumos performance with encrypted filesystems and aes-ni comparable to other platforms?
Unfortunately this is still very relevant; for my particular test system, it's 90% slower.
I recently ran a series of tests on different zfs pool and dataset configurations on an EPYC 3101 (Zen 1, 4 core). For the most part it was able to sustain near-disk speeds of ~1800 MiB/s, until I got to encryption: any of the zfs encryption options hit a CPU-bound limit of ~200 MiB/s. I got a 10% improvement by disabling use of AES-NI instructions, indicating the current KCF is spending far more time managing blocks than it is doing any actual work.
There have been significant changes to the CTR code since Sašo's original patch [1], so it will require some effort to bring it up to mergeable state. OpenZFS started with a port of KCF, but discovered the same problem and independently made their own improvements in 2020 [2].
Under the same test conditions, FreeBSD has no trouble maintaining near-disk speeds with AES GCM. (CCM can only manage ~430 MiB/s, although I suspect most will not find that mode to be of practical interest.)
[1] https://github.com/illumos/illumos-gate/compare/master...skiselkov:illumos-gate:4896_aes_fast.patch
[2] https://github.com/openzfs/zfs/pull/9749