International Business Machines Corporation

Globalfoundries Inc.

Ety Khaitzin

Danny Harnik

Sergey Marenkov

Dmitry Sotnikov

David Darden Chambliss

Joseph S Glider

Yosef Shatsky

Jonathan Fischer-Toubol

Ronen Itshak Kat

Jonathan Amit

Roger C Raphael

Asaf Porat-Stoler

Afief Halumi

Shai Ishaya Tahar

Reut Cohen

Tom Sivan

Chaim Koifman

Scott Schumacher

Ramasuri Narayanam

Rajesh M Desai

Ben Sasson

Moshe Weiss

Nir Halowani

Aviv Caro

Arjun Natarajan

Vladimir Shalikashvili

Michael Moser

Amir Epstein

Rishi Saket

Effi Ofer

Yoni Raveh

Yifat Kuttner

Angineh Aghakiant

Ritwik Chaudhuri

Roee Shlomo

Rohith Dwarakanath Vallam

Gil Shapira

Maya Anderson

Shalu Agrawal

Ohad Eytan

A mechanism is provided for dispersed location-based data storage. A request is received to write a data file to a referrer memory region in a set of memory regions. For each data chunk of the data file, responsive to a comparison of a hash value for the data chunk to other hash values for other stored data chunks referenced in the referrer memory region indicating that the data chunk fails to exist in the referrer memory region, responsive to the data chunk existing in another memory region in the set of memory regions, responsive to the memory region failing to be one of the predetermined number N of owner memory regions associated with the referrer memory region, and responsive to the predetermined number N of owner memory regions failing to have been met, a reference to the data chunk is stored in the referrer memory region.

Performance of dispersed location-based deduplication

One embodiment provides a method, including: receiving, at a central system, a query requesting access to a dataset, wherein the central system communicates with a plurality of data storage locations, each having a governance policy for data stored at the data storage location, wherein different portions of the dataset are stored within different of the plurality of data storage locations; sending a sub-query formulated based upon the query; receiving a governance enforcement actions listing corresponding to the portion of the dataset stored within the corresponding data storage location; generating a meta-policy of enforcement actions for all of the plurality of data storage locations storing portions of the dataset, wherein the meta-policy identifies enforcement actions and an order of the enforcement actions to be applied to the dataset; and providing the meta-policy to each of the plurality of data storage locations.

Consistent governance with asset constraints across data storage locations

Embodiments of the present systems and methods may provide a data access approval process that supports complex and fine-grained policies and can be applied to different data items at scale, which provides improvement over current technologies. For example, in an embodiment, a computer-implemented method for controlling access to data by computer systems may comprise generating an intermediate representation by integrating a combination of data access policies, data attributes including attributes per data subject, and the data itself to form the intermediate representation, receiving a request for access to the data, rewriting the request for access to the data to incorporate the intermediate representation so as to provide access only to data allowed by the policies integrated into the intermediate representation, and executing the rewritten request and providing only data allowed by the policies integrated into the intermediate representation.

Fine-grained privacy enforcement and policy-based data access control at scale

A computer-implemented method according to one embodiment includes identifying a plurality of storage systems within a storage environment, determining characteristics of each of the plurality of storage systems, the characteristics including one or more data reduction techniques implemented by each of the plurality of storage systems, performing a plurality of storage simulations of one or more data volumes, utilizing the characteristics of each of the plurality of storage systems, and determining one of the plurality of storage systems to store the one or more data volumes, based on results of the plurality of storage simulations.

Distributing data across a mixed data storage center

A method for managing a data cache, comprising: storing a cache management list comprising a plurality of entries and having: a tail part stored in a first storage and documenting recently accessed data items stored in the data cache, a body part stored in a second storage and documenting less recently accessed data items stored in the data cache, and a head part stored in the first storage and documenting least recently accessed data items stored in the data cache; and in each of a plurality of iterations: receiving at least one data access request; documenting the data access request in the tail; identifying a plurality of duplicated entries present in the body and the tail; and removing each of the plurality of duplicated entries from the body in the second storage according to a physical organization in the second storage of the plurality of duplicated entries.

Managing a least-recently-used data cache with a persistent body

Enforcement of policies for tabular data access as a collection of columns over a plurality of different information assets is provided. In an enforcement knowledge graph, information asset-assigned terms are found that correspond to information assets in a virtual information asset that references a set of tabular data. Transitive closures of the information asset-assigned terms are found in a business glossary to form a table of business glossary terms. Term intersection is determined between a hash table of any column-assigned terms and the table of business glossary terms. The information assets are assigned to the virtual information asset when the term intersection is not empty. A set of policy rules associated with the set of tabular data and a context of a user making a data access request to the set of tabular data is applied to the virtual information asset to determine an access enforcement decision.

Dynamic enforcement of data protection policies for arbitrary tabular data access to a corpus of rectangular data sets

A method, apparatus, system, and computer program product for evaluating enforcement decisions on an asset using a policy. Rules in the policy are applied by a computer system to the asset taking into account a context for a request to access the asset in response receiving to the request to access the asset, and wherein the rules in the policy determine whether access to the asset is allowed. A determination is made by the computer system as to whether a conflict is present in an initial decision made using the rules in the policy. A set of conflict resolution processes are applied by the computer system when the conflict is present such that a final decision is made on the request to access the asset.

Policy rule enforcement decision evaluation with conflict resolution

Computer program products, as well as corresponding systems and methods are configured for performing deduplication in conjunction with random read and write operations, and include: computing a fingerprint of data included in a write request; determining whether a short term dictionary comprises an entry corresponding to the fingerprint; in response to determining the short term dictionary comprises the entry corresponding to the fingerprint, writing the data to a data store in a deduplicating manner; in response to determining the short term dictionary does not comprise the entry, determining whether a long term dictionary corresponding to the namespace comprises the entry; in response to determining the long term dictionary comprises the entry, writing the data to the data store in the deduplicating manner; and in response to determining the long term dictionary does not comprise the entry, writing the data to the data store in a non-deduplicating manner.

Region-integrated data deduplication implementing a multi-lifetime duplicate finder

In one embodiment, a deduplicating storage system includes a processor and logic integrated with and/or executable by the processor. The logic is configured to cause the processor to perform a method which includes: computing a fingerprint of a data chunk, and determining whether a short term dictionary corresponding to the namespace comprises an entry corresponding to the fingerprint. In response to determining the short term dictionary does not comprise the entry, a determination is made whether a long term dictionary corresponding to the namespace comprises the entry. In response to determining the long term dictionary comprises the entry: the data chunk is written to the data store in the deduplicating manner, and the short term dictionary is repopulated with the entry. Moreover, in response to determining the long term dictionary does not comprise the entry, the data chunk is written to the data store in a non-deduplicating manner.

A computer-implemented method includes dividing a data set into a plurality of regions and dividing the plurality of regions into a plurality of chunks of fixed size. The computer-implemented method further includes determining a sample size of the plurality of chunks to be sampled for each region, wherein the sample size is determined based, at least in part, on an acceptance of a likelihood of identifying at least one collision between two regions corresponding to logical entities of a first cluster of logical entities. The computer-implemented method further includes sampling the plurality of chunks for each region based on the determined sample size. The computer-implemented method further includes generating a hash value for each chunk sampled and storing each hash value in an index. The computer-implemented method further includes identifying one or more collisions between the plurality of regions. A corresponding computer system and computer program product are also disclosed.

Identification of high deduplication data

A computer-implemented method includes receiving a set of basis fingerprints corresponding to image chunks within a basis set of image regions wherein each image region within the basis set of image regions comprises one or more image chunks, and generating a fingerprint for each image chunk of a plurality of selected image chunks within an unprocessed region of a machine image to produce a plurality of sampled fingerprints. The method also includes determining a similarity metric for the unprocessed region from the sampled fingerprints and the basis fingerprints, comparing the similarity metric for the unprocessed region with a selected threshold, and including the unprocessed region within the basis set of image regions in response to determining that the similarity metric is less than the selected threshold. A corresponding computer program product and computer system are also disclosed herein.

Deduplication ratio estimation using an expandable basis set

Methods, computing systems and computer program products implement embodiments of the present invention that include partitioning a dataset into a full set of logical data units, and selecting a sample subset of the full set, the sample subset including a random sample of the full set based on a sampling ratio. A set of target hash values are selected from a full range of hash values, and, using a hash function, a respective unit hash value is calculated for each of the logical data units in the sample subset. A histogram is computed that indicates a duplication count of each of the unit hash values that matches a given target hash value, and based on the histogram, a number of distinct logical data units in the full set is estimated.

Low memory sampling-based estimation of distinct elements and deduplication

A computer-implemented method includes receiving a set of representative machine image regions for a computing environment wherein the set of representative machine image regions collectively comprise a set of representative image chunks. The method also includes generating a fingerprint for each representative image chunk within the set of representative image chunks to produce a set of representative fingerprints, generating a fingerprint for selected image chunks within a measured machine image region to produce a set of sampled fingerprints, and determining a deduplication metric for the measured machine image region based on the representative fingerprints and the sampled fingerprints. A corresponding computer program product and computer system are also disclosed herein.

Deduplication ratio estimation using representative images

Growing community of inventors

Holon, Israel

Ety Khaitzin