Project

General

Profile

Feature #7953

Support for the Zstandard compression

Added by r a over 2 years ago. Updated over 2 years ago.

Status:
New
Priority:
Low
Assignee:
-
Category:
-
Target version:
-
Start date:
2017-03-08
Due date:
% Done:

0%

Estimated time:
Difficulty:
Medium
Tags:
needs-triage

Description

Zstandard (Zstd) is a new real time lossless compression library providing high compression ratios.

The website is here http://facebook.github.io/zstd/

Performance information taken from http://fastcompression.blogspot.co.uk/2015/01/zstd-stronger-compression-algorithm.html

Name                Ratio  C.speed D.speed
                                           MB/s    MB/s
    zlib 1.2.8 -6        3.099     18      275
    zstd                   2.872    201      498
    zlib 1.2.8 -1        2.730     58      250
    LZ4 HC r127       2.720     26     1720
    QuickLZ 1.5.1b6 2.237    323       373
    LZO 2.06           2.106    351       510
    Snappy 1.1.0      2.091    238      964
    LZ4 r127            2.084    370    1590
    LZF 3.6              2.077    220      502

Can this be considered for inclusion in a future release of OpenIndiana and ZFS

History

#1

Updated by Predrag Zečević over 2 years ago

Hi,
I did some testing and managed it to compile (with some tweaks):

 wget https://github.com/facebook/zstd/archive/v1.1.2.tar.gz
 mv v1.1.2.tar.gz zstd-1.1.2.tar.gz
 tar xfv zstd-1.1.2.tar.gz
 cd zstd-1.1.2/

 export CC=gcc
 export CXX=g++

 sed -i 's#OpenBSD FreeBSD DragonFly NetBSD#OpenBSD FreeBSD DragonFly NetBSD SunOS#g' $(find . -type f -name Makefile)
 sed -i 's#/usr/local#/opt/SFW#g' $(find . -type f -name Makefile)

 pfexec gmake install

 cd contrib/pzstd
 pfexec gmake install

If I manage to get some time, I could add to OI github.
$ zstd -h
*** zstd command line interface 32-bits v1.1.2, by Yann Collet ***
Usage :
      zstd [args] [FILE(s)] [-o file]

FILE    : a filename
          with no FILE, or when FILE is - , read standard input
Arguments :
 -#     : # compression level (1-19, default:3) 
 -d     : decompression 
 -D file: use `file` as Dictionary 
 -o file: result stored into `file` (only if 1 input file) 
 -f     : overwrite output without prompting 
--rm    : remove source file(s) after successful de/compression 
 -k     : preserve source file(s) (default) 
 -h/-H  : display help/long help and exit

Advanced arguments :
 -V     : display Version number and exit
 -v     : verbose mode; specify multiple times to increase log level (default:2)
 -q     : suppress warnings; specify twice to suppress errors too
 -c     : force write to standard output, even if it is the console
 -r     : operate recursively on directories
--ultra : enable levels beyond 19, up to 22 (requires more memory)
--no-dictID : don't write dictID into header (dictionary compression)
--[no-]check : integrity check (default:enabled)
--test  : test compressed file integrity 
--[no-]sparse : sparse mode (default:enabled on file, disabled on stdout)
 -M#    : Set a memory usage limit for decompression 
--      : All arguments after "--" are treated as files 

Dictionary builder :
--train ## : create a dictionary from a training set of files 
 -o file : `file` is dictionary name (default: dictionary) 
--maxdict ## : limit dictionary to specified size (default : 112640) 
 -s#    : dictionary selectivity level (default: 9)
--dictID ## : force dictionary ID to specified value (default: random)

Benchmark arguments :
 -b#    : benchmark file(s), using # compression level (default : 1) 
 -e#    : test all compression levels from -bX to # (default: 1)
 -i#    : minimum evaluation time in seconds (default : 3s)
 -B#    : cut file into independent blocks of size # (default: no block)


$ pzstd -h
Usage:
  pzstd [args] [FILE(s)]
Parallel ZSTD options:
  -p, --processes   #    : number of threads to use for (de)compression (default:4)
ZSTD options:
  -#                     : # compression level (1-19, default:3)
  -d, --decompress       : decompression
  -o                file : result stored into `file` (only if 1 input file)
  -f, --force            : overwrite output without prompting
      --rm               : remove source file(s) after successful (de)compression
  -k, --keep             : preserve source file(s) (default)
  -h, --help             : display help and exit
  -V, --version          : display version number and exit
  -v, --verbose          : verbose mode; specify multiple times to increase log level (default:2)
  -q, --quiet            : suppress warnings; specify twice to suppress errors too
  -c, --stdout           : force write to standard output, even if it is the console
  -r                     : operate recursively on directories
      --ultra            : enable levels beyond 19, up to 22 (requires more memory)
  -C, --check            : integrity check (default)
      --no-check         : no integrity check
  -t, --test             : test compressed file integrity
  --                     : all arguments after "--" are treated as files


$ zstd -19 solaris_containers.pdf
solaris_containers.pdf : 89.35%   (446779 => 399178 bytes, solaris_containers.pdf.zst)

Also available in: Atom PDF