Feature #3890
open
New deduplication hash algorithm
0%
Description
Hello guys,
do you think it's possible to add a new deduplication hash algoritm less cpu intensive?
I mean sha 256 is really nice because you have a very very low probability of hash collision but in real world it's just to cpu consumming to use it.
Why don't implement another algorithm like sha-1 or md5? Plus you have the verify mode if a collision appear.
Updated by Patrick Domack almost 10 years ago
This is why there is fletcher hashing available, and at a normal rate of gigabytes per second per cpu core, it's more than fast enough.
support for sha256 will be changing quick enough, as sha was designed for speed only in hardware based chips.
Updated by Anthony Glidic over 9 years ago
Hello that is not true,
actually fletcher4 is only usable for checksumming not for deduplication.
Today the only algorithm for deduplication is sha256 (and by the way when deduplication is on you should turn off checksumming, it's useless to calcul 2 hashs).
with a lower algorithm like sha1 (160 bits) it will lower the cpu load reduce the hash size so reduce the global size of ddt and kept a good security.
P.S: i talk about sha1 but i'm sure other new and better algorithm exist for that