International Business Machines Corporation

Globalfoundries Inc.

Chaim Koifman

Jonathan Amit

Avi Zohar

Nadav Kedem

Sergey Marenkov

Dmitry Sotnikov

Nir Halowani

Lilia Demidov

Rostislav Raikhman

George Goldberg

Ronen Itshak Kat

Danny Harnik

Yosef Shatsky

Ori Shalev

Aviv Caro

David Darden Chambliss

Joseph S Glider

Shai Ishaya Tahar

Orit Nissan-Messing

Ben Sasson

Oded Margalit

Ety Khaitzin

Jonathan Fischer-Toubol

Amir Lidor

Lior Shlomov

Assaf Yaari

Kat I Ronen

Nir Friedman

Shai Taharlev

Methods, computing systems and computer program products implement embodiments of the present invention that include configuring a storage system to store multiple storage entities for access by one or more host computers in communication with the storage system, and specifying a compression condition including a minimum compression ratio. The storage system can then estimate an expected compression ratio for a given storage entity, compress the given storage entity upon the expected compression ratio meeting the compression condition, and provide, to a given host computer, access to the compressed given storage entity.

Storage data reduction analysis and forecast

A Huffman cache is used to hold Huffman dictionaries that are changeable for dynamically selecting literal frequencies that are similar, wherein the Huffman cache is a data storage cache.

Efficient caching of Huffman dictionaries

Identification of data candidates for data processing is performed in real time by a processor device in a computing environment. Data candidates are sampled for performing a classification-based compression upon the data candidates. A heuristic is computed on a randomly selected data sample from the data candidate, the heuristic computed by, for each one of the data classes, calculating an expected number of characters to be in a data class, calculating an expected number of characters that will not belong to a predefined set of the data classes, and calculating an actual number of the characters for each of the data classes and the non-classifiable data.

Real-time identification of data candidates for classification based compression

Systems and methods for transforming a logical data object for storage in a storage device operable with at least one storage protocol, creating, reading, writing, optimization and restoring thereof. Transforming the logical data object comprises creating in the storage device a transformed logical data object comprising and one or more allocated storage sections with a predefined size; processing one or more sequentially obtained chunks corresponding to the transforming logical data object, wherein at least one of said processed data chunks comprises transformed data resulting from said processing; sequentially storing the processed data chunks into said storage sections in accordance with an order the chunks received. The method further includes reading a data range from the transformed logical object in response to a read request specifying desired point in time to be read.

Systems and methods for transformation of logical data objects for storage

Systems and methods for compressing a raw logical data object () for storage in a storage device operable with at least one storage protocol, creating, reading, writing, optimizatic in and restoring thereof. Compressing the raw logical data object () comprises creating in the storage device a compressed logical data object () comprising a header () and one or more allocated compressed sections with predefined size (---); compressing one or more sequentially obtained chunks of raw data (---) corresponding to the raw logical data object () thus giving rise to the compressed data chunks (---); and sequentially accommodating the processed data chunks into: said compressed sections (---) in accordance with an order said chunks received, wherein said compressed sections serve as atomic elements of compression/decompression operations during input/output transactions on the logical data object.

In one embodiment, a computer-implemented method for performing deduplication in conjunction with random read and write operations across a namespace that is divided into a plurality of disjoint regions is disclosed. The method includes: maintaining a metadata structure for each of the plurality of disjoint regions via the respective region manager, each metadata structure comprising metadata indicating a physical storage location of one or more data chunks associated with the respective region; and performing a deduplicated write operation. The deduplicated write operation includes: determining whether the disjoint regions comprise a second data chunk identical in content to a first data chunk; and in response to determining one or more of the plurality of disjoint regions other than the first region includes the second data chunk, establishing a reference from the first data chunk to the second data chunk. Corresponding systems and computer program products are also disclosed.

Region-integrated data deduplication

In one embodiment, a computer program product is configured for performing deduplication in conjunction with random read and write operations across a namespace divided into a plurality of disjoint regions. The computer program product includes a computer readable storage medium having program instructions embodied therewith, where the computer readable storage medium is not a transitory signal per se. The program instructions are executable by a computer to cause the computer to perform a method including: maintaining a metadata structure for each of the plurality of disjoint regions via the respective region manager(s) of the plurality of disjoint regions, each metadata structure comprising metadata indicating a physical storage location of one or more data chunks associated with the respective region; and performing, by the computer, a deduplicated write operation of a first data chunk in a first region of the plurality of disjoint regions.

Compression blocks are divided into partitions creating a two dimensional divide of the compression blocks by slicing the compression blocks forming a first dimension and sub-partitioning the compression blocks into the partitions forming a second dimension. Each one of the partitions are compressed in separate compression streams.

Data compression using compression blocks and partitions

For real-time classification of data into data compression domains, a decision is made for which of the data compression domains write operations should be forwarded by reading randomly selected data of the write operations for computing a set of classifying heuristics thereby creating a fingerprint for each of the write operations. The write operations having a similar fingerprint are compressed together in a similar compression stream.

Real-time classification of data into data compression domains

Calculating fingerprints for each one of a multiplicity of alignment combinations of fixed-size deduplication data blocks and comparing each of the fingerprints to stored deduplicated data fingerprints in a lookup database for determining a preferred deduplication data block alignment. A deduplication data block comprises each of the fixed-size deduplication data blocks.

Lookup-based data block alignment for data deduplication

Various embodiments for data compression by a processor. Levels of data distribution are configured for data processing, including a first level of the data distribution incorporating a GRID network of data storage nodes, and a second level of the data distribution incorporating a GRID network of compressive nodes in communication with the GRID network of data storage nodes. Input/output (I/O) for an associated storage volume is load balanced between the data storage nodes, as data passes through the first level into the second level to be compressed or uncompressed.

Three dimensional grid compression storage

Identification of data candidates for data processing is performed in real time by a processor device in a distributed computing environment. Data candidates are sampled for performing a classification-based compression upon the data candidates. A heuristic is computed on a randomly selected data sample from the data candidate, the heuristic computed by, for each one of the data classes, calculating an expected number of characters to be in a data class, calculating an expected number of characters that will not belong to a predefined set of the data classes, and calculating an actual number of the characters for each of the data classes and the non-classifiable data.

Methods and systems for compressing a logical data object for storage in a storage device in a distributed network are provided. One method includes creating, in the storage device, a compressed logical data object including a header and one or more allocated compressed sections with a predefined size, compressing the one or more obtained chunks of data corresponding to the logical data object, thus giving rise to the compressed data chunks, and storing the compressed data chunks in the compressed sections, wherein the compressed sections serve as atomic elements of compression or decompression operations during input/output transactions on the logical data object.

Systems and methods for transforming a logical data object for storage in a storage device operable with at least one storage protocol, creating, reading, writing, optimization and restoring thereof. Transforming the logical data object comprises creating in the storage device a transformed logical data object comprising one or more allocated storage sections with a predefined size; transforming one or more sequentially obtained chunks of obtained data corresponding to the transforming logical data object; and sequentially storing the processed data chunks into said storage sections in accordance with a receive order of said chunks, wherein said storage sections serve as atomic elements of transformation/de-transformation operations during input/output transactions on the logical data object. The processing may comprise two or more data transformation techniques coordinated in time, concurrently executing autonomous sets of instructions, and provided in a manner preserving the sequence of processing and storing the processed data chunks.

Growing community of inventors

Rishon Lezion, Israel

Chaim Koifman