Feature #7953
Support for the Zstandard compression
Status:
New
Priority:
Low
Assignee:
-
Category:
-
Target version:
-
Start date:
2017-03-08
Due date:
% Done:
0%
Estimated time:
Difficulty:
Medium
Tags:
needs-triage
Description
Zstandard (Zstd) is a new real time lossless compression library providing high compression ratios.
The website is here http://facebook.github.io/zstd/
Performance information taken from http://fastcompression.blogspot.co.uk/2015/01/zstd-stronger-compression-algorithm.html
Name Ratio C.speed D.speed
MB/s MB/s
zlib 1.2.8 -6 3.099 18 275
zstd 2.872 201 498
zlib 1.2.8 -1 2.730 58 250
LZ4 HC r127 2.720 26 1720
QuickLZ 1.5.1b6 2.237 323 373
LZO 2.06 2.106 351 510
Snappy 1.1.0 2.091 238 964
LZ4 r127 2.084 370 1590
LZF 3.6 2.077 220 502
Can this be considered for inclusion in a future release of OpenIndiana and ZFS
Updated by Predrag Zečević almost 4 years ago
Hi,
I did some testing and managed it to compile (with some tweaks):
wget https://github.com/facebook/zstd/archive/v1.1.2.tar.gz mv v1.1.2.tar.gz zstd-1.1.2.tar.gz tar xfv zstd-1.1.2.tar.gz cd zstd-1.1.2/ export CC=gcc export CXX=g++ sed -i 's#OpenBSD FreeBSD DragonFly NetBSD#OpenBSD FreeBSD DragonFly NetBSD SunOS#g' $(find . -type f -name Makefile) sed -i 's#/usr/local#/opt/SFW#g' $(find . -type f -name Makefile) pfexec gmake install cd contrib/pzstd pfexec gmake install
If I manage to get some time, I could add to OI github.
$ zstd -h *** zstd command line interface 32-bits v1.1.2, by Yann Collet *** Usage : zstd [args] [FILE(s)] [-o file] FILE : a filename with no FILE, or when FILE is - , read standard input Arguments : -# : # compression level (1-19, default:3) -d : decompression -D file: use `file` as Dictionary -o file: result stored into `file` (only if 1 input file) -f : overwrite output without prompting --rm : remove source file(s) after successful de/compression -k : preserve source file(s) (default) -h/-H : display help/long help and exit Advanced arguments : -V : display Version number and exit -v : verbose mode; specify multiple times to increase log level (default:2) -q : suppress warnings; specify twice to suppress errors too -c : force write to standard output, even if it is the console -r : operate recursively on directories --ultra : enable levels beyond 19, up to 22 (requires more memory) --no-dictID : don't write dictID into header (dictionary compression) --[no-]check : integrity check (default:enabled) --test : test compressed file integrity --[no-]sparse : sparse mode (default:enabled on file, disabled on stdout) -M# : Set a memory usage limit for decompression -- : All arguments after "--" are treated as files Dictionary builder : --train ## : create a dictionary from a training set of files -o file : `file` is dictionary name (default: dictionary) --maxdict ## : limit dictionary to specified size (default : 112640) -s# : dictionary selectivity level (default: 9) --dictID ## : force dictionary ID to specified value (default: random) Benchmark arguments : -b# : benchmark file(s), using # compression level (default : 1) -e# : test all compression levels from -bX to # (default: 1) -i# : minimum evaluation time in seconds (default : 3s) -B# : cut file into independent blocks of size # (default: no block)
$ pzstd -h Usage: pzstd [args] [FILE(s)] Parallel ZSTD options: -p, --processes # : number of threads to use for (de)compression (default:4) ZSTD options: -# : # compression level (1-19, default:3) -d, --decompress : decompression -o file : result stored into `file` (only if 1 input file) -f, --force : overwrite output without prompting --rm : remove source file(s) after successful (de)compression -k, --keep : preserve source file(s) (default) -h, --help : display help and exit -V, --version : display version number and exit -v, --verbose : verbose mode; specify multiple times to increase log level (default:2) -q, --quiet : suppress warnings; specify twice to suppress errors too -c, --stdout : force write to standard output, even if it is the console -r : operate recursively on directories --ultra : enable levels beyond 19, up to 22 (requires more memory) -C, --check : integrity check (default) --no-check : no integrity check -t, --test : test compressed file integrity -- : all arguments after "--" are treated as files
$ zstd -19 solaris_containers.pdf solaris_containers.pdf : 89.35% (446779 => 399178 bytes, solaris_containers.pdf.zst)