Project

General

Profile

Feature #7953

Support for the Zstandard compression

Added by r a over 2 years ago. Updated over 2 years ago.

Status:
New
Priority:
Low
Assignee:
-
Category:
-
Target version:
-
Start date:
2017-03-08
Due date:
% Done:

0%

Estimated time:
Difficulty:
Medium
Tags:
needs-triage

Description

Zstandard (Zstd) is a new real time lossless compression library providing high compression ratios.

The website is here http://facebook.github.io/zstd/

Performance information taken from http://fastcompression.blogspot.co.uk/2015/01/zstd-stronger-compression-algorithm.html

Name                Ratio  C.speed D.speed
MB/s MB/s
zlib 1.2.8 -6 3.099 18 275
zstd 2.872 201 498
zlib 1.2.8 -1 2.730 58 250
LZ4 HC r127 2.720 26 1720
QuickLZ 1.5.1b6 2.237 323 373
LZO 2.06 2.106 351 510
Snappy 1.1.0 2.091 238 964
LZ4 r127 2.084 370 1590
LZF 3.6 2.077 220 502

Can this be considered for inclusion in a future release of OpenIndiana and ZFS

History

#1

Updated by Predrag Zečević over 2 years ago

Hi,
I did some testing and managed it to compile (with some tweaks):

 wget https://github.com/facebook/zstd/archive/v1.1.2.tar.gz
 mv v1.1.2.tar.gz zstd-1.1.2.tar.gz
 tar xfv zstd-1.1.2.tar.gz
 cd zstd-1.1.2/

 export CC=gcc
 export CXX=g++

 sed -i 's#OpenBSD FreeBSD DragonFly NetBSD#OpenBSD FreeBSD DragonFly NetBSD SunOS#g' $(find . -type f -name Makefile)
 sed -i 's#/usr/local#/opt/SFW#g' $(find . -type f -name Makefile)

 pfexec gmake install

 cd contrib/pzstd
 pfexec gmake install

If I manage to get some time, I could add to OI github.
$ zstd -h
*** zstd command line interface 32-bits v1.1.2, by Yann Collet ***
Usage :
      zstd [args] [FILE(s)] [-o file]

FILE    : a filename
          with no FILE, or when FILE is - , read standard input
Arguments :
 -#     : # compression level (1-19, default:3) 
 -d     : decompression 
 -D file: use `file` as Dictionary 
 -o file: result stored into `file` (only if 1 input file) 
 -f     : overwrite output without prompting 
--rm    : remove source file(s) after successful de/compression 
 -k     : preserve source file(s) (default) 
 -h/-H  : display help/long help and exit

Advanced arguments :
 -V     : display Version number and exit
 -v     : verbose mode; specify multiple times to increase log level (default:2)
 -q     : suppress warnings; specify twice to suppress errors too
 -c     : force write to standard output, even if it is the console
 -r     : operate recursively on directories
--ultra : enable levels beyond 19, up to 22 (requires more memory)
--no-dictID : don't write dictID into header (dictionary compression)
--[no-]check : integrity check (default:enabled)
--test  : test compressed file integrity 
--[no-]sparse : sparse mode (default:enabled on file, disabled on stdout)
 -M#    : Set a memory usage limit for decompression 
--      : All arguments after "--" are treated as files 

Dictionary builder :
--train ## : create a dictionary from a training set of files 
 -o file : `file` is dictionary name (default: dictionary) 
--maxdict ## : limit dictionary to specified size (default : 112640) 
 -s#    : dictionary selectivity level (default: 9)
--dictID ## : force dictionary ID to specified value (default: random)

Benchmark arguments :
 -b#    : benchmark file(s), using # compression level (default : 1) 
 -e#    : test all compression levels from -bX to # (default: 1)
 -i#    : minimum evaluation time in seconds (default : 3s)
 -B#    : cut file into independent blocks of size # (default: no block)

$ pzstd -h
Usage:
  pzstd [args] [FILE(s)]
Parallel ZSTD options:
  -p, --processes   #    : number of threads to use for (de)compression (default:4)
ZSTD options:
  -#                     : # compression level (1-19, default:3)
  -d, --decompress       : decompression
  -o                file : result stored into `file` (only if 1 input file)
  -f, --force            : overwrite output without prompting
      --rm               : remove source file(s) after successful (de)compression
  -k, --keep             : preserve source file(s) (default)
  -h, --help             : display help and exit
  -V, --version          : display version number and exit
  -v, --verbose          : verbose mode; specify multiple times to increase log level (default:2)
  -q, --quiet            : suppress warnings; specify twice to suppress errors too
  -c, --stdout           : force write to standard output, even if it is the console
  -r                     : operate recursively on directories
      --ultra            : enable levels beyond 19, up to 22 (requires more memory)
  -C, --check            : integrity check (default)
      --no-check         : no integrity check
  -t, --test             : test compressed file integrity
  --                     : all arguments after "--" are treated as files

$ zstd -19 solaris_containers.pdf
solaris_containers.pdf : 89.35%   (446779 => 399178 bytes, solaris_containers.pdf.zst) 

Also available in: Atom PDF