International Business Machines Corporation

Heiko Schloesser

Erik Rueger

Frank Krick

Christof Schmitt

Mathias Defiebre

Itzhack Goldberg

Neil Sondhi

Ezra Hartz

A data de-duplication system includes a storage device that includes multiple data sets, a fixed read-only storage area, and a processing device configured to perform certain functions. The system determines whether each of the multiple data sets has met a common pattern criteria, and if so, identifies the data set as a candidate data set for de-duplication. The common pattern criteria is indicative of whether a chunk data is frequently accessed among multiple users on a cloud. The system stores data in at least one candidate data sets in the fixed storage area. For each of the at least candidate data sets, the system generates a unique pointer that corresponds to a location of the at least one candidate data sets in the fixed storage area. The system further uses the pointers to de-duplicate the at least one candidate data sets.

System and method for storage optimization in a de-duplication system

A data stream is stored in storage media. As part of the storage, the data stream is divided into a plurality of chunks. The plurality of chunks include a target chunk that is next to a first chunk in a file within the data stream. A determination is made that the target chunk matches an existing chunk stored in the storage media. In response to the determination, a first pointer to the existing stored chunk is created in file metadata for the file. Also in response to the determination, a second pointer to a first stored chunk that matches the first chunk is created in chunk metadata embedded with the existing stored chunk.

Metadata recovery for de-duplicated data

A technique for operating a data storage system includes accessing respective absolute expiration times for valid pages in one or more candidate storage blocks. Garbage collection is performed on ones of the candidate storage blocks that include a first predetermined number of the valid pages that do not expire, as indicated by the absolute expiration times, prior to a first predetermined time period. Garbage collection is deferred on ones of the candidate storage blocks that include a second predetermined number of the valid pages that expire, as indicated by the absolute expiration times, prior to a second predetermined time period.

Techniques for selecting storage blocks for garbage collection based on longevity information

A mechanism is provided for opportunistic handling of freed data in data de-duplication. Responsive to receiving a request to store a file in a storage device, the file is mapped to a set of virtual blocks. For each virtual block in the set of virtual blocks: a hash value is computed, a determination is made as to whether the computed hash value appears within a previously-used information table as associated with an existing data block, and, responsive to the computed hash value appearing within a previously-used information table as associated with an existing data block, a data block entry and hash value associated with the existing data block is moved to a de-duplication information table. The virtual block is then stored as a reference to the existing data block.

Growing community of inventors

Tucson, AZ, United States of America

Heiko Schloesser