File Semantic Aware Primary Storage Deduplication System

Abstract

The deployment of the primary storage deduplication system at the file level is a daunting challenge due to disk-bottleneck and data fragmentation problems. File semantics such as type and size can be used to reduce deduplication overhead. There is negligible data redundancy across different types of files compared to the same type of files. Hence, the application of the same deduplication method irrespective of file type causes wastage of computing resources. In this paper, the File Aware DeDuplicaton system is proposed. Files are partitioned based on the size as small and large files. Large files are categorized based on the data redundancy as high, low, and unpredictable file types. File type-specific deduplication approach is applied for each category of files. Separate index tables are maintained for each type and hash table for small files of all types. The FADD system is simulated in the Linux environment using two different types of FIU traces and some locally collected data sets. The effectiveness of FADD is compared with the full deduplication system and Hybrid Deduplication System based on the parameters – metadata access overhead, average segment length, and response time. The experimental results show that the system has performed consistently better for all input data sets.

Keywords:

• Data fragmentation
• Deduplication
• Disk bottleneck

DISCLOSURE STATEMENT

No potential conflict of interest was reported by the author(s).

Additional information

Notes on contributors

Amdewar Godavari

Amdewar Godavari is currently working toward the PhD degree in the Department of Computer Science and Engineering at National Institute of Technology, Warangal, India. Her research interests includes distributed systems, data deduplication, and cache system.

Chapram Sudhakar

Chapram Sudhakar is currently working as associate professor in the Department of Computer Science and Engineering at National Institute of Technology, Warangal, India. His research interests include cloud computing, distributed systems, high performance computing, operating systems.Email: [email protected]

T. Ramesh

T Ramesh is presently working as visiting professor in the Department of Computer Science and Engineering at National Institute of Technology, Warangal, India. His research interests includes theoretical computer science and distributed systems.Email: [email protected]