International Business Machines Corporation

Shira Ben-Dor

Lior Aronovich

Ofer Leneman

Shay Haim Akirav

Michael Hirsch

Joseph W Dain

Asaf Levy

Gil E Paz

Aviv Caro

Elena Drobchenko

Tzafrir Z Taub

Dan Melamed

Liran Loya

Samuel Krikler

Amir Kredi

Oded Aviyam

Aviad Zlotnick

Yariv Bachar

Orit Nissan-Messing

Rafael Buchbinder

Harry Butterworth

Adam Wolman

Various embodiments for repository management in a data deduplication system, by a processor device, are provided. Metadata of an inode structure of an entire pre-allocated file system is captured, exported, and compressed from an existing deduplication appliance, the pre-allocated file system comprising a fully padded file system. The exported and compressed metadata of the pre-allocated file system is decompressed and imported into a data deduplication repository of a new deduplication appliance having an identical file system size as within the existing deduplication appliance, to initially configure or subsequently scale the inode structure of a file system of the data deduplication repository of the new deduplication appliance efficiently.

Streamlined padding of deduplication repository file systems

Various embodiments for managing data in a data storage having data deduplication. A back reference data structure is configured for user data segments as a mechanism to identify an affected storage block to which information in the back reference data structure refers. The back reference data structure is initialized such that a resolution of the back reference data structure diminishes as a number of the user data segments referencing the affected storage block increases.

Back referencing of deduplicated data

Various embodiments for repository management in a data deduplication system, by a processor device, are provided. Metadata of a pre-allocated file system is captured and exported. The exported metadata is then imported into a data deduplication repository for configuring the data deduplication repository with minimum overhead.

For efficient calculation of both similarity search values and boundaries of digest blocks in data deduplication, input data is partitioned into chunks, and for each chunk a set of rolling hash values is calculated. A single linear scan of the rolling hash values is used to produce both similarity search values and boundaries of the digest blocks of the chunk. The rolling hash values are used to contribute to the calculation of the similarity search values and to the calculation of the boundaries of the digest blocks.

Efficient calculation of similarity search values and digest block boundaries for data deduplication

For digest retrieval based on similarity search in deduplication processing in a data deduplication system using a processor device in a computing environment, input data is partitioned into fixed sized data chunks. Similarity elements and digest block boundaries and digest values are calculated for each of the fixed sized data chunks. Matching similarity elements are searched for in a search structure containing the similarity elements for each of the fixed sized data chunks in a repository of data. Positions of similar data are located in the repository. The positions of the similar data are used to locate and load into the memory stored digest values and corresponding stored digest block boundaries of the similar data in the repository. The digest values and the corresponding digest block boundaries of the input data are matched with the stored digest values and the corresponding stored digest block boundaries to find data matches.

Digest retrieval based on similarity search in data deduplication

For efficient calculation of both similarity search values and boundaries of digest blocks in data deduplication, input data is partitioned into chunks, and for each chunk a set of rolling hash values is calculated. A single linear scan of the rolling hash values is used to produce both similarity search values and boundaries of the digest blocks of the chunk.

A deduplication storage capacity is estimated as a function of an expected deduplication ratio, the expected deduplication ratio being a combined average of a current deduplication ratio and a configured deduplication ratio, the current deduplication ratio depending on the data currently stored in the deduplication storage, and the configured deduplication ratio being an estimate made at a configuration stage of the deduplication computing storage environment.

Capacity management in deduplication storage systems

A data storage system including at least one storage controller having a first color policy and operative to store data onto a first data storage unit at a primary site as part of a current color of the primary site, at least one storage controller having a second color policy and operative to store data onto a second data storage unit at the primary site as part of the current color, and a color control node operative to provide each of the controllers with new color information while maintaining the integrity of dependent writes across color boundaries.

Growing community of inventors

Giv′at-Shmuel, Israel

Shira Ben-Dor