International Business Machines Corporation

Emc Corporation

Michael Hirsch

Lior Aronovich

Yair Toaff

Shmuel Tomi Klein

Ron Asher

Shay Haim Akirav

Haim Bitner

Eitan Bachmat

Ofer Leneman

Ariel J Ish-Shalom

Ehud Meiri

Itzhack Goldberg

Neil Sondhi

Ron Edelstein

Liran Loya

Shira Ben-Dor

Thomas S Klein

Itai Tzur

Yuval Ofek

Frank Krick

Dan Arnon

Thorsten Krause

Samuel Krikler

Zoran Cakeljic

Christophe Fagiano

Sharon Galtzur

Shmuel Thomas Klein

Michael Edward Factor

Peter Kamvysselis

Ronen Itshak Kat

Nadav Kedem

Deduplicated data is packed into finite-sized containers. A similarity score is calculated between files that are similarly of the deduplicated data. The similarity score is used for grouping the similarly compared files of the deduplicated data into subsets for destaging each of the subsets from a deduplication system to one a finite-sized container. The similarity score is used for grouping the similarly compared files of the deduplicated data into subsets for destaging each of the subsets from a deduplication system to one of the finite-sized containers. An indication is received by a user of which of the similarly compared files are to be grouped into the subsets for destaging each of the subsets from a deduplication system to one of the finite-sized containers. Transitive closures are used for assisting with using the similarity score for grouping the similarly compared files of the deduplicated data into the subsets.

Packing deduplicated data into finite-sized containers

For adding cooperative file coloring protocols in a data deduplication system using a processor device in a computing environment, a preferred character is represented for file coloring in a file using a code selected from a multiplicity of codes that represent a variety of contexts. The original meaning of the preferred character is retained when representing the preferred character for the file coloring by the code selected from the multiplicity of codes. The file is deduplicated by the data deduplication system according to the file coloring that represents a source file of a backup application.

Adding cooperative file coloring protocols in a data deduplication system

Methods, computer systems, and computer program products for processing data a computing environment are provided. The computer environment for data deduplication storage receives a plurality of write operations for deduplication storage of the data. The data is buffered in a plurality of buffers with overflow temporarily stored to a memory hierarchy when the data received for deduplication storage is sequential or non sequential. The data is accumulated and updated in the plurality of buffers per a data structure, the data structure serving as a fragment map between the plurality of buffers and a plurality of user file locations. The data is restructured in the plurality of buffers to form a complete sequence of a required sequence size. The data is provided as at least one stream to a stream-based deduplication algorithm for processing and storage.

Method and system for processing data

A computer-implemented method, according to one embodiment includes, for each repository data chunk in repository data that comprises a plurality of the repository data chunks, generating a corresponding set of repository distinguishing characteristics (RDCs). Each set of RDCs is generated by: applying a hash function to the respective input data chunk or repository data chunk to generate a plurality of hashes, each hash comprising a hash value and a hash position within the data chunk, applying a first function to the plurality of generated hashes to identify a first subset of hashes distributed across the data chunk, applying a second function to the hash positions of the hashes of the first subset to identify a second subset of the plurality of generated hashes, and defining the second subset of hashes as the set of RDCs.

Systems and methods for efficient data searching, storage and reduction

Input backup data is deduplicated with data of a synthetic backup previously constructed by a deduplication storage system. A synthetic backup is constructed by processing metadata instructions provided by a backup application. Deduplication digests are calculated based on the data of the synthetic backup and the deduplication digests are stored in a digests index. When new backup data is processed, deduplication digests of the new data are calculated and searched in the digests index. Matching digests of previously constructed synthetic backups are located in the digests index. Each of the located matching digest references stored data are included in the synthetic backup, and the stored data is similar to the input backup data. Data matches are found in the input backup data and data in the synthetic backup.

Deduplicating input backup data with data of a synthetic backup previously constructed by a deduplication storage system

Creation of synthetic backups within deduplication storage system by a backup application

For read ahead of digests in similarity based data deduplication in a data deduplication system using a processor device in a computing environment, input data is partitioned into data chunks and digest values are calculated for each of the data chunks. The positions and sizes of similar data intervals in a repository of data are found for each of the data chunks. The positions and the sizes of read ahead intervals are calculated based on the similar data intervals. The read ahead digests of the read ahead intervals are located and loaded into memory in a background read ahead process.

Read ahead of digests in similarity based data deduplicaton

A computer program product for searching a repository of binary uninterpretted data, according to one embodiment, includes a computer readable storage medium having program instructions executable by a computer to cause the computer to perform a method comprising: analyzing, by the computer, segments of each of the repository and input data to determine a repository segment that is similar to an input segment, the analyzing including searching an index of representation values of the repository data for matching representation values of the input in a time independent of a size of the repository and linear in a size of the input data; and analyzing, by the computer, the similar repository segment with respect to the input segment to determine their common data sections while utilizing at least some of the matching representation values for data alignment, in a time linear in a size of the input segment.

For producing digest block segmentations based on reference segmentations in a data deduplication system using a processor device in a computing environment, digests are calculated for an input data chunk. Data matches and data mismatches are produced based on matching input digests with reference digests. Secondary digest block segmentations are obtained from similar reference intervals for each of the data mismatches and applied to the input data.

Digest block segmentation based on reference segmentation in a data deduplication system

For self-healing in a hash-based deduplication system using a processor device in a computing environment, deduplication digests of data and a corresponding list of the deduplication digests in a table of contents (TOC) are maintained for the self-healing of data that is lost or unreadable. The input data digests are compared to the TOC if directed to data that is lost or unreadable, and the input data digests are used to repair the one of lost and unreadable data.

Self-healing by hash-based deduplication

Deduplicated data is packed into finite-sized containers. A similarity score is calculated between files that are similarly of the deduplicated data. The similarity score is used for grouping the similarly compared files of the deduplicated data into subsets for destaging each of the subsets from a deduplication system to one a finite-sized container.

A deduplication storage system and a backup application create a synthetic backup. Metadata instructions are provided to the deduplication storage system. Each of the metadata instructions specifies the data segment of an originating backup and a designated location of the data segment in the synthetic backup. A set of metadata instructions is transformed into a transformed set of metadata instructions.

For adding cooperative file coloring protocols in a data deduplication system using a processor device in a computing environment, a preferred character is represented for file coloring in a file using a code selected from a multiplicity of codes that represent a variety of contexts. The original meaning of the preferred character is retained when representing the preferred character for the file coloring by the code selected from the multiplicity of codes.

Growing community of inventors

Mazkeret Batya, Israel

Michael Hirsch