Project

General

Profile

Feature #3890

New deduplication hash algorithm

Added by Anthony Glidic almost 7 years ago. Updated over 6 years ago.

Status:
New
Priority:
Normal
Assignee:
-
Category:
-
Start date:
2013-07-16
Due date:
% Done:

0%

Estimated time:
Difficulty:
Medium
Tags:
needs-triage
Gerrit CR:

Description

Hello guys,
do you think it's possible to add a new deduplication hash algoritm less cpu intensive?

I mean sha 256 is really nice because you have a very very low probability of hash collision but in real world it's just to cpu consumming to use it.
Why don't implement another algorithm like sha-1 or md5? Plus you have the verify mode if a collision appear.

History

#1

Updated by Patrick Domack almost 7 years ago

This is why there is fletcher hashing available, and at a normal rate of gigabytes per second per cpu core, it's more than fast enough.

support for sha256 will be changing quick enough, as sha was designed for speed only in hardware based chips.

#2

Updated by Anthony Glidic over 6 years ago

Hello that is not true,

actually fletcher4 is only usable for checksumming not for deduplication.

Today the only algorithm for deduplication is sha256 (and by the way when deduplication is on you should turn off checksumming, it's useless to calcul 2 hashs).

with a lower algorithm like sha1 (160 bits) it will lower the cpu load reduce the hash size so reduce the global size of ddt and kept a good security.

P.S: i talk about sha1 but i'm sure other new and better algorithm exist for that

Also available in: Atom PDF