What is deduplication in Big Data?

Writen by Devil 10:38 PM - 165 Comments

Deduplication or Dedupe in simple words is the process were data blocks are analysed to identify duplicate blocks and the system stores only one copy of it while deleting the rest.

 Big Data / Cloud storage vendors like Google, Microsoft and Amazon are constantly seeking ways by which their customers can manage expanding silos of data into storage devices. 

Currently two major technologies are : Compression & Deduplication

follow for more on -> Compression: 

It helps to minimise the size of a file by eliminating unnecessary data from the file.

Types of Data Deduplication

• Source Deduplication : 

Source deduplication is the removal of redundancies from data before transmission to the backup target. 

It uses the client software for comparing new data blocks on the primary storage device with the previously backed up data blocks.

• Target Deduplication 

Target deduplication removes all the redundant data in the backup appliance most often on virtual tape library or a NAS device. It reduces the storage capacity required for backup data but does not reduces the amount of data sent across LAN or WAN.

• Inline Deduplication 

Inline deduplication is the removal of redundancies from data before or as it is being written to a backup device. Inline deduplication reduces the amount of redundant data in an application and the capacity needed for the backup disk targets.

• Post-Process Deduplication 

Post-process deduplication writes the backup data into the disk cache before it starts the dedupe process. It is mostly used in the backup applications, virtual tape libraries and the like, where reduction of backup time is required.

• Global Deduplication

Global data deduplication is a method of preventing redundant data when backing up data to multiple deduplication devices. It removes all the possible backup data redundancies across multiple systems.

• Facebook
• Tweeter
• Google+
• Whatsapp Pinterest