International Business Machines Corporation

Ronen Itshak Kat

Sergey Marenkov

Danny Harnik

Dmitry Sotnikov

George Goldberg

Jonathan Amit

Michael Edward Factor

Chaim Koifman

Nir Halowani

Lilia Demidov

Ety Khaitzin

Kalman Zvi Meth

Amir Epstein

Miriam Allalouf

Roee Shlomo

Itzhack Goldberg

Neil Sondhi

Michael Hirsch

Elliot Karl Kolodner

Julian Satran

David Blair Whitworth

Nitzan Peleg

Dalit Naor

Lee Charles LaFrese

Moshe Klausner

Sima Nadler

Yaron Weinsberg

Moshik Hershcovitch

Shlomit Koyfman

Grigori Chtrasberg

Doron Chen

Maya Anderson

Roger Kahn

An example system includes a processor to generate a data plane based on dataset metadata, data plane component metadata, data governance decisions, and information technology (IT) system metrics. The processor is to also apply components of the data plane for a specified workload across a number of computing devices selected by a control plane.

Generating data planes for processing of data workloads

Embodiments of the present systems and methods may provide a data access approval process that supports complex and fine-grained policies and can be applied to different data items at scale, which provides improvement over current technologies. For example, in an embodiment, a computer-implemented method for controlling access to data by computer systems may comprise generating an intermediate representation by integrating a combination of data access policies, data attributes including attributes per data subject, and the data itself to form the intermediate representation, receiving a request for access to the data, rewriting the request for access to the data to incorporate the intermediate representation so as to provide access only to data allowed by the policies integrated into the intermediate representation, and executing the rewritten request and providing only data allowed by the policies integrated into the intermediate representation.

Fine-grained privacy enforcement and policy-based data access control at scale

In some examples, a system for executing instructions can include a processor to detect data to be transmitted to a storage device in response to a write operation. The processor can also determine that the data comprises a compressible characteristic that enables compression of the data to a size below a threshold value. Additionally, the processor can generate a modified data block by encrypting the compressed data, and adding a padding to the compressed and encrypted data. Furthermore, the processor can transmit the modified data block to the storage device.

Encrypted and compressed data transmission with padding

A computer-implemented method includes receiving a set of basis fingerprints corresponding to image chunks within a basis set of image regions wherein each image region within the basis set of image regions comprises one or more image chunks, and generating a fingerprint for each image chunk of a plurality of selected image chunks within an unprocessed region of a machine image to produce a plurality of sampled fingerprints. The method also includes determining a similarity metric for the unprocessed region from the sampled fingerprints and the basis fingerprints, comparing the similarity metric for the unprocessed region with a selected threshold, and including the unprocessed region within the basis set of image regions in response to determining that the similarity metric is less than the selected threshold. A corresponding computer program product and computer system are also disclosed herein.

Deduplication ratio estimation using an expandable basis set

A system to reduce the amount of storage and memory used to maintain derived datasets is disclosed. The system operates by using pointers to the underlying data in persistent, byte-addressable storage media. The system additionally reduces the creation time of the views when storage class memory (SCM) is the underlying storage. Furthermore, the invention relates to a form of compression that is tailored to the use cases of big data analytics. The processes of this disclosure use random access to significantly improve performance.

Method for efficient derived data views using byte addressable persistent storage

A method for a data placement that attempts to predict the most suitable placement, in terms of data reduction, of a newly created storage volume based on the volumes known attributes and the current placement of volumes to deduplication domains is disclosed. The system uses machine learning to perform improved deduplication-aware placement. The system attempts to predict the deduplication domain where a newly created volume would eventually have the best content sharing. The system does this by using the known attributes of the volume at the time of creation, such as owner, volume name, initial size, creation time, and the history of data already in the system and its placement.

Cognitive deduplication-aware data placement in large scale storage systems

Identification of data candidates for data processing is performed in real time by a processor device in a computing environment. Data candidates are sampled for performing a classification-based compression upon the data candidates. A heuristic is computed on a randomly selected data sample from the data candidate, the heuristic computed by, for each one of the data classes, calculating an expected number of characters to be in a data class, calculating an expected number of characters that will not belong to a predefined set of the data classes, and calculating an actual number of the characters for each of the data classes and the non-classifiable data.

Real-time identification of data candidates for classification based compression

A method for storage systems improvement includes collecting information that indicates one or more failure correlations for disks in a storage system. The disks are then separated into a plurality of virtual failure domains based on the indicated one or more failure correlations. The method then determines that all data objects of a set of redundant data objects are included in a first virtual failure domain. Responsive to determining that all data objects of the set of redundant data objects are included in the first virtual failure domain, the method then migrates at least one data object of the set of redundant data objects from a first disk in the first virtual failure domain to a second disk in a second virtual failure domain.

Virtual failure domains for storage systems

Identification of data candidates for data processing is performed in real time by a processor device in a distributed computing environment. Data candidates are sampled for performing a classification-based compression upon the data candidates. A heuristic is computed on a randomly selected data sample from the data candidate, the heuristic computed by, for each one of the data classes, calculating an expected number of characters to be in a data class, calculating an expected number of characters that will not belong to a predefined set of the data classes, and calculating an actual number of the characters for each of the data classes and the non-classifiable data.

Machines, systems and methods for increasing data resiliency in a computing system, the method comprising distinguishing between first data and second data stored in one or more data storage mediums, wherein the first data is more vulnerable than the second data for the purpose of recovering lost data; and recovering the first data before recovering the second data. Increasing redundancy protection for the first data to increase chances for data recovery by way of data reconstruction; and decreasing redundancy protection for the first data, after the first data has been backed up at least once.

Data recovery scheme based on data backup status

Identification of data candidates for data processing is performed in real time by a processor device in a computing environment. Data candidates are sampled for performing a classification-based compression upon the data candidates. A heuristic is computed on a randomly selected data sample from the data candidate for determining if the data candidate may benefit from the classification-based compression, wherein a ratio is summed between the actual number of the characters and the expected number of the characters, and then dividing the ratio by a number of the data classes that are not empty, wherein the non-classifiable data are included in the number of the data classes during the dividing, and the number of the data classes, that are not empty, have characters that belong to the class that were observed in the input; and the classification-based compression is performed on the data candidates if the ratio exceeds a threshold.

Batch processing machines, systems and methods are provided. The method comprises copying target data from a target location to a first data storage medium, wherein the target data stored in the first data storage medium is utilized by a batch process; executing the batch process; checkpointing state information for the batch process at one or more time intervals while the batch process is executing; determining whether a data record in the target data stored in the target location was updated while the batch process was executing; and reinstating the batch process from a point in time prior to an earliest point in the batch process when a representation of an updated data record in the target data in the target location was accessed by the batch process.

Restarting a batch process from an execution point

Identification of data candidates for data processing is performed in real time by a processor device in a computing environment. Data candidates are sampled for performing a classification-based compression upon the data candidates. A heuristic is computed on a randomly selected data sample from the data candidate for determining if the data candidate may benefit from the classification-based compression. A decision is provided for approving the classification-based compression on the data candidates according to the heuristic.

Growing community of inventors

Kokav-Yair, Israel

Ronen Itshak Kat