Study on Data De-Duplication Based on Bloom Filter in Fault Tolerant Erasure Codes

Erasure codes have become a mainstream fault-tolerant method in the field of cloud storage fault tolerance. Erasure codes will be the data from the file level storage into a block level storage, there will be a lot of repeat encoding block. Data De-duplication technique provide an effective solution to the minimization of storage. Bloom filter is a kind of space efficient data structure, which is suitable for the query in finite space. To solve the problem of erasure codes repetition coding block, it proposes a global bloom filter array efficient methods, and through the dynamic expansion of the standard Bloom filters and set up a global dynamic bloom filter array GDBFA, so as to effectively control the system false positive rate. Experiments show that: GDBFA has the characteristics of high De-duplication rate, low false positive rate, and less influence on the performance of fault tolerance.