International Business Machines Corporation

Globalfoundries Inc.

Nir Halowani

Lilia Demidov

Jonathan Amit

Sergey Marenkov

Chaim Koifman

Dmitry Sotnikov

George Goldberg

Ronen Itshak Kat

Ben Sasson

Ori Shalev

Yifat Kuttner

Danny Harnik

Shai Ishaya Tahar

Yakov Gerlovin

Oded Margalit

Ety Khaitzin

Jonathan Fischer-Toubol

Kat I Ronen

Gil Shapira

A Huffman cache is used to hold Huffman dictionaries that are changeable for dynamically selecting literal frequencies that are similar, wherein the Huffman cache is a data storage cache.

Efficient caching of Huffman dictionaries

Identification of data candidates for data processing is performed in real time by a processor device in a computing environment. Data candidates are sampled for performing a classification-based compression upon the data candidates. A heuristic is computed on a randomly selected data sample from the data candidate, the heuristic computed by, for each one of the data classes, calculating an expected number of characters to be in a data class, calculating an expected number of characters that will not belong to a predefined set of the data classes, and calculating an actual number of the characters for each of the data classes and the non-classifiable data.

Real-time identification of data candidates for classification based compression

B-Tree data is serialized to existing data for all types of workloads. Each of an identified data range is encoded with frequency encoding, wherein a first value in a frequency encoded identified data range is a first value in original data and all subsequent values in the frequency encoded identified data range are equal to a difference between a corresponding value in an input file and a previous value in the input file.

Compression of serialized B-tree data

B-Tree data is serialized to existing data for all types of workloads by converting a B-Tree data structure into a format capable of being stored and resurrected while containing all data stored in the B-Tree data structure and information relating to the B-Tree data structure. The serialized B-Tree data is divided into a plurality of sections. The serialized B-Tree data is stored into a plurality of buffers, where storing the B-Tree information section in a first binary buffer, the B-Tree key section in a second binary buffer, and the B-Tree data section in a third binary buffer. In the B-Tree data section, B-Tree data elements stored in the B-Tree data structure are saved, where a size of the B-Tree data section is equal to a total number of the B-Tree data elements in the B-Tree data structure multiplied by a size of each of the B-Tree data elements.

Efficient B-Tree data serialization

A detection learning module is used for enabling and/or disabling real-time compression detection by maintaining a history of real-time compression detection success for sampled data. The enabling or disabling of the real-time compression detection is based on a detection benefit function derived from a set of calculated heuristics indicating the real-time compression detection success on input streams. The detection benefit function is calculated based on at least one heuristic score.

Controlling real-time compression detection

Compression blocks are divided into partitions creating a two dimensional divide of the compression blocks by slicing the compression blocks forming a first dimension and sub-partitioning the compression blocks into the partitions forming a second dimension. Each one of the partitions are compressed in separate compression streams.

Data compression using compression blocks and partitions

For real-time classification of data into data compression domains, a decision is made for which of the data compression domains write operations should be forwarded by reading randomly selected data of the write operations for computing a set of classifying heuristics thereby creating a fingerprint for each of the write operations. The write operations having a similar fingerprint are compressed together in a similar compression stream.

Real-time classification of data into data compression domains

Identification of data candidates for data processing is performed in real time by a processor device in a distributed computing environment. Data candidates are sampled for performing a classification-based compression upon the data candidates. A heuristic is computed on a randomly selected data sample from the data candidate, the heuristic computed by, for each one of the data classes, calculating an expected number of characters to be in a data class, calculating an expected number of characters that will not belong to a predefined set of the data classes, and calculating an actual number of the characters for each of the data classes and the non-classifiable data.

Data is converted into a minimized data representation using a suffix tree by sorting data streams according to symbolic representations for building table boundary formation patterns. The converted data is fully reversible for reconstruction while retaining minimal header information. A scanning operation is performed by searching a suffix of each of the sorted data streams for identifying a data sequence that includes a first symbol representing textual data, and a second symbol representing numerical data. The suffix tree for the converted data is then built.

Table boundary detection in data blocks for compression

B-Tree data is serialized to existing data for all types of workloads. The serialized B-Tree data, that has been split, sorted and classified into identified data ranges, is then compressed.

B-Tree data is serialized to existing data for all types of workloads by converting a B-Tree data structure into a format capable of being stored and resurrected while containing all data stored in the B-Tree data structure and information relating to the B-Tree data structure.

Efficient B-tree data serialization

Exemplary method, system, and computer program product embodiments for efficient one-pass cache-aware compression are provided. In one embodiment, by way of example only, an output of a fast compressor to Huffman encoding for achieving the one-pass cache-aware compression by using a predetermined Huffman-tree upon determining by the fast compressor a final representation of each data byte.

Efficient one-pass cache-aware compression

Identification of data candidates for data processing is performed in real time by a processor device in a computing environment. Data candidates are sampled for performing a classification-based compression upon the data candidates. A heuristic is computed on a randomly selected data sample from the data candidate for determining if the data candidate may benefit from the classification-based compression, wherein a ratio is summed between the actual number of the characters and the expected number of the characters, and then dividing the ratio by a number of the data classes that are not empty, wherein the non-classifiable data are included in the number of the data classes during the dividing, and the number of the data classes, that are not empty, have characters that belong to the class that were observed in the input; and the classification-based compression is performed on the data candidates if the ratio exceeds a threshold.

A detection learning module is used for enabling and/or disabling real-time compression detection by maintaining a history of real-time compression detection success for sampled data. The enabling or disabling of the real-time compression detection is based on a detection benefit function derived from a set of calculated heuristics indicating the real-time compression detection success on input streams.

Growing community of inventors

Holon, Israel

Nir Halowani