Cluster-based distributed dynamic cuckoo filter system for Redis

ABSTRACT

With the exponential growth of network data storage scale, the issue of uniform distribution and efficient retrieval of data in the distributed storage systems such as the Redis cluster has received increasing attention in recent years. In view of the existing problems in scalability, usability and other aspects of the solution in current researches, we propose the distributed dynamic cuckoo filter system based on Redis cluster. On one hand, we introduce an efficient hash indexing structure–dynamic cuckoo filter (DCF), which only stores the fingerprint information of data, and has the automatically scalable capacity to meet the demand of data storage on a dynamic scale. On the other hand, we use an improved consistent hashing algorithm to construct Redis cluster and use the thorough communication mechanism of Redis cluster to achieve the data sharing and efficient utilisation of multi-machine filters. The scheme proposed in this paper can take the time and space efficiency into account, greatly promote the retrieval performance of massive data, and improve the reliability and availability of Redis cluster.

KEYWORDS:

• Distributed storage system
• Redis cluster
• dynamic cuckoo filter
• multi-machine

ORCID

Peng Li http://orcid.org/0000-0001-5026-5347

Additional information

Funding

The subject is sponsored by the National Key R&D Program of China [grant number 2018YFB1003201]; the National Natural Science Foundation of P. R. China [grant number 61672296], [grant number 61602261]; Major Natural Science Research Projects in Colleges and Universities of Jiangsu Province [grant number 18KJA520008].