Deduplication

Kirth A. Prawl

Prof. Eli Weissman

Devry College of New York

October 10, 2012

Deduplication technology is a way to streamline backup and storage of data by making sure that redundant records are filtered out prior to backup. In some cases, using deduplication technology can eliminate as much as 95% of the data traditionally backed up. If that figure seems high, consider the case of corporate documents that are distributed to hundreds or even thousands of employees. Each individual may have an identical copy of electronic files such as procedures manuals.
When all workstations are backed up without considering duplicate records, the backup itself will contain much wasted space. By managing the data more efficiently and identifying files that are identical, deduplication technology can produce a backup that is scaled down in size, making it more manageable in terms of time and resources as well as in terms of physical media such as disk or tape space.
Deduplication technology can be implemented at the network level and consist of only local operations, or it can also be integrated into cloud based technologies that take advantage of economies of scale through the use of remote and managed services.
Deduplicate data Eliminating redundant data can significantly shrink storage requirements and improve bandwidth efficiency. Because primary storage has gotten cheaper over time, enterprises typically store many versions of the same information so that new work can reuse old work. Some operations like Backup store extremely dismissed information. Deduplication lowers storage costs since fewer disks are needed, and shortens backup/recovery times since there can be far less data to transfer. In the context of backup and other near line data, we can make a strong supposition that there is a great deal of duplicate data. The same data keeps getting stored over and over again consuming a lot of unnecessary storage space disk or tape, electricity to power and cool the disk or tape drives, and bandwidth for replication, creating a chain of cost and resource inefficiencies within the organization.
The way that Deduplication works, that the Deduplication segments the incoming data stream, uniquely identifies the data segments, and then compares the segments to previously stored data. If an incoming data segment is a duplicate of what has already been stored, the segment is not stored again, but a reference is created to it. If the segment is unique, it is stored on disk.
For example, if a file or volume that is backed up every week creates a significant amount of duplicate data. Deduplication algorithms analyze the data and can store only the compressed, unique change elements of that file. This process can provide an average of 10-30 times or greater reduction in storage capacity requirements, with average backup retention policies on normal enterprise data. This means that many companies can store 10TB to 30TB of backup data on 1 TB of physical disk capacity, which has huge economic benefits.
The Virtual tape library data deduplication technology is implemented around a VTL, the capabilities of the VTL must be considered as part of the evaluation process. It is unlikely that the savings from data deduplication will override the difficulties caused by using a sub-standard VTL. Consider the functionality, performance, stability and support of the VTL as well as its deduplication extension.
The Impact of deduplication on backup performance is important to consider where and when data deduplication takes place in relation to the backup process. Although some solutions attempt deduplication while data is being backed up, this straight method processes the backup stream as it comes into the deduplication application, making performance dependent on the single node’s strength. Such an approach can slow down backups, jeopardize backup windows and degrade VTL performance over time. For maximum manageability, the solution should allow for granular tape or group level policy based deduplication based on a variety of factors resource operation, production schedules, time since creation and so on. In this way, storage efficiencies can be achieved while optimizing the use of system resources.
Scalability is the solution is being chosen for longer term data storage, scalability, in terms of both capacity and performance, is an important reflection. Consider how much data you will want to keep on disk for fast access over the next five years. How will the data index system scale to your requirements. A deduplication solution should provide an architecture that allows economic right sizing for both the initial implementation and the long term growth of the system. Gathering allows organizations to scale to meet growing capacity requirements even for environments with many petabytes of data without compromising deduplication efficiency or system performance. Clustering also inherently provides a high availability environment, protecting the backup repository interface VTL or file interface and deduplication nodes by offering failover support.
Data deduplication delivers benefits throughout a distributed enterprise, not just in a single data center. A solution that includes replication and multiple levels of deduplication can achieve maximum benefits from the technology. The solution should only require minimal bandwidth for the central site to determine whether the remote data is contained in the central repository. Only unique data across all sites should be replicated to the central site and subsequently to the disaster recovery site, to avoid excessive bandwidth requirements.
It is extremely important to create a highly available deduplication source. Since a very large amount of data has been combined in one location, risk tolerance for data loss is very low. Access to the deduplicated data source is critical and should not be vulnerable to a single point of failure. The solution should have fail over capability in the event of a node failure. Even if multiple nodes in a group fail, the company must be able to continue to recover its data and respond to the business.
The best efficiency and effectiveness way to keep file based deduplication approaches do not reduce storage capacity requirements as much as those that analyze data at a sub file or block level. Consider, for example, changing a single line in a 4 megabyte performance. In a file based solution, the entire file must be stored, doubling the storage required. If the demonstration is sent to multiple people, as presentations often are, the negative effects multiply. Most sub file deduplication processes use some sort of “chunking” method to break up a large amount of data, such as a virtual tape cartridge, into smaller sized pieces to search for duplicate data. Larger chunks of data can be processed at a faster rate, but less duplication is detected.
Based on my conclusion Deduplication technology can help you avoid the costs associated with building massive storage arrays. You can choose to use deduplication to keep your filers hard drives from breaking your budget, for keeping your backup sets from requiring a forklift to transport or both. Choose the type of deduplication very wisely source, target or inline and then implement it carefully choosing the chunk sizes and hashing algorithms among other concerns. If you do, you’ll allow your company to smoothly grow with its storage demands and retain important information for longer periods of time.

Work Citied
Books:
Data Deduplication: High Impact Strategies by Kevin Roebuck
Guide to Data Deduplication by Larry Freeman
Pro Data Backup and Recovry by Steven Nelson

Website: http://www.computerweekly.com/feature/How-data-deduplication-works http://technet.microsoft.com/en-us/library/hh831602.aspx