International Business Machines Corporation

Yair Toaff

Lior Aronovich

Michael Hirsch

Ron Asher

Gil E Paz

Shmuel Tomi Klein

Ehud Meiri

Thomas S Klein

Shmuel Thomas Klein

Shay Haim Akirav

Input backup data is deduplicated with data of a synthetic backup previously constructed by a deduplication storage system. A synthetic backup is constructed by processing metadata instructions provided by a backup application. Deduplication digests are calculated based on the data of the synthetic backup and the deduplication digests are stored in a digests index. When new backup data is processed, deduplication digests of the new data are calculated and searched in the digests index. Matching digests of previously constructed synthetic backups are located in the digests index. Each of the located matching digest references stored data are included in the synthetic backup, and the stored data is similar to the input backup data. Data matches are found in the input backup data and data in the synthetic backup.

Deduplicating input backup data with data of a synthetic backup previously constructed by a deduplication storage system

Creation of synthetic backups within deduplication storage system by a backup application

For producing digest block segmentations based on reference segmentations in a data deduplication system using a processor device in a computing environment, digests are calculated for an input data chunk. Data matches and data mismatches are produced based on matching input digests with reference digests. Secondary digest block segmentations are obtained from similar reference intervals for each of the data mismatches and applied to the input data.

Digest block segmentation based on reference segmentation in a data deduplication system

A deduplication storage system and a backup application create a synthetic backup. Metadata instructions are provided to the deduplication storage system. Each of the metadata instructions specifies the data segment of an originating backup and a designated location of the data segment in the synthetic backup. A set of metadata instructions is transformed into a transformed set of metadata instructions.

Exemplary method, system, and computer program product embodiments for scalable data deduplication working with small data chunk in a computing environment are provided. In one embodiment, by way of example only, for each small data chunk, a signature is generated based on a combination of a representation of characters used in selecting data to be deduplicated. A c-spectrum of the small data chunk being a sequence of representations of different characters ordered by a frequency of occurrence in the small data chunk, and an f-spectrum of the small data chunk being a corresponding sequence of frequencies of the different characters in the small data chunk.

Scalable deduplication system with small blocks

A deduplication storage system and a backup application create a synthetic backup. Metadata instructions are provided to the deduplication storage system. Each of the metadata instructions specifies the data segment of an originating backup and a designated location of the data segment in the synthetic backup. Each of the metadata instructions are processed by locating those data sub-segments in the deduplication storage system specified by the data segment in each of the metadata instructions, and creating metadata references to each of the data sub-segments and adding the metadata references to metadata of the synthetic backup being created.

Creation of synthetic backups within deduplication storage system

Segment sizes are controlled by setting the size of a segment boundary in a hash-based backup deduplication system in a distributed computing environment. A subsequence of size K of a sequence of characters S is set. Segment boundaries are set by using the sequence of the decreasingly restrictive logical tests if one of the sequence of the decreasingly restrictive logical tests returns a true value when applied on the sequence of characters S.

Controlling segment size distribution in hash-based deduplication

Segment sizes are controlled by setting the size of a segment boundary in a hash-based deduplication system. A subsequence of size K of a sequence of characters S is set. Segment boundaries are set by using the sequence of the decreasingly restrictive logical tests if one of the sequence of the decreasingly restrictive logical tests returns a true value when applied on the sequence of characters S.

Various embodiments are provided for managing a global cache coherency in a distributed shared caching for a clustered file system (CFS). The CFS manages access permissions to an entire space of data segments by using the DSM module. In response to receiving a request to access one of the data segments, a calculation operation is performed for obtaining most recent contents of one of the data segments. The calculation operation performs one of providing the most recent contents via communication with a remote DSM module which obtains the one of the data segments from an associated external cache memory, instructing by the DSM module to read from storage the one of the data segments, and determining that any existing contents of the one of the data segments in the local external cache are the most recent contents.

Managing global cache coherency in a distributed shared caching for clustered file systems

Input backup data is deduplicated with data of a synthetic backup previously constructed by a deduplication storage. A synthetic backup is constructed by processing metadata instructions provided by a backup application. Deduplication digests are calculated based on the data of the synthetic backup and the deduplication digests are stored in a digests index. When new backup data is processed, deduplication digests of the new data are calculated and searched in the digests index. A data segment of the synthetic backup is partitioned into fixed sized sub-segments. The calculated digests of sub-segment are aggregated to produce the deduplication digest, and the deduplication digest is formed for the synthetic backup.

Calculating deduplication digests for a synthetic backup by a deduplication storage system

Various embodiments are provided for managing a global cache coherency in a distributed shared caching for a clustered file system (CFS). The CFS manages access permissions to an entire space of data segments by using the DSM module. In response to receiving a request to access one of the data segments, a calculation operation is performed for obtaining most recent contents of one of the data segments. Most recent contents are determined if ownership of the one of the data segments is possessed by a remote DSM module and the request to access one of the data segments is for shared permission and exists in the local external cache. The most recent contents are transported within the response if the response is in a remote external cache and has a valid permission for the one of the data segments otherwise reading from the one of the data segments.

Exemplary method, system, and computer program product embodiments for full exploitation of parallel processors for data processing are provided. In one embodiment, by way of example only, a set of parallel processors is partitioned into disjoint subsets according to indices of the set of the parallel processors. The size of each of the disjoint subsets corresponds to a number of processors assigned to the processing of the data chunks at one of the layers. A transition function is devised from the indices of the set of the parallel processors at one time steps to the indices of the set of the parallel processors at a following time step.

Full exploitation of parallel processors for data processing

Exemplary method, system, and computer program product embodiments for full exploitation of parallel processors for data processing are provided. In one embodiment, by way of example only, a set of parallel processors is partitioned into disjoint subsets according to indices of the set of the parallel processors. The size of each of the disjoint subsets corresponds to a number of processors assigned to the processing of the data chunks at one of the layers. Each of the processors are assigned to different layers in different data chunks such that each of processors are busy and the data chunks are fully processed within a number of the time steps equal to the number of the layers. A transition function is devised from the indices of the set of the parallel processors at one time steps to the indices of the set of the parallel processors at a following time step.

Methods, computer systems, and computer program products for calculating a remainder by division of a sequence of bytes interpreted as a first number by a second number are provided. A first subset of bytes is read, and an associated first remainder by division is calculated and stored in the memory location from which the subset was read. A second subset of bytes is read, and an associated second remainder by division is calculated with a second processor. The calculating of the second remainder by division may occur at least partially during the calculating of the first remainder by division. A third and fourth subset of bytes is read and associated remainders are calculated.

Growing community of inventors

Giv′at-Shmuel, Israel

Yair Toaff