International Business Machines Corporation

Gil Vernik

Michael Edward Factor

Elliot Karl Kolodner

Danny Harnik

Dmitry Sotnikov

Dalit Naor

Oded Margalit

Eran Rom

Embodiments of the present systems and methods may provide the capability ensure that data is persisted and accessed correctly without depending on eventually consistent list operations on the object store. For example, in an embodiment, a computer-implemented method for data distribution may comprise attempting to persist a plurality of data parts from a plurality of processing tasks, generating a manifest including information indicating those attempts to persist data parts that have succeeded, and persisting the manifest with the data parts that have been successfully persisted.

Persisting distributed data sets into eventually consistent storage systems

Embodiments of the present systems and methods may provide techniques to provide simple and accurate estimate of memory requirements for application invocation in a serverless environment. For example, a method may comprise selecting sample invocations of functions as a service from a larger plurality of invocations, submitting for execution the plurality of sample invocations and, for each sample invocation, submitting a specification of a memory size to be used for execution of each sample invocation, determining, whether the specification of the memory size to be used for execution of each sample invocation results in unsuccessful execution of at least some of the sample invocations due to insufficient memory and, if so, adjusting the specification of the memory size for at least some of the sample invocations, and submitting for execution at least those invocations in the larger plurality of invocations that were not included in the plurality of sample invocations.

Dynamic determination of memory requirements for function as a service multi-invocation flows

Aspects of the present invention include a method, system and computer program product for performing data deduplication for eventually consistent distributed data storage (DDS) system. The method includes receiving data content from one or more clients by a DDS system, wherein the one or more clients do not coordinate transmitting of the data content. The method also includes calculating a hash for the data content by the distributed data storage system, writing the data content to an object used for data deduplication, wherein a name of the object is based on the hash and determining whether the data content is present in the distributed data storage system based on the name of an object previously stored on the DDS system. The method further includes keeping track of a number of references to the data content and delaying deletion of the data content for a predetermined period of time.

Data deduplication for an eventually consistent system

A computer-implemented method for determining whether to perform a pushdown may include receiving a request for analytics to be performed by an analytics platform on data stored in bulk storage. An operation may be identified as a candidate for a pushdown, where the operation is selected from among one or more operations to be performed for fulfilling the request. The pushdown would require the operation to be performed at the bulk storage. The data may be sampled by reading one or more samples of the data, where the one or more samples are a fraction of the data. The operation may be performed, by a computer processor, on the one or more samples. It may be determined, based on performing the operation on the one or more samples, whether to perform the pushdown of the operation.

Selective operation pushdowns from an analytics platform to bulk storage

Machines, systems and methods of uploading data files, the method comprising a first client machine dividing a first file into N data chunks to be uploaded to a server, wherein the N data chunks are of size kX, where k is an integer and X is size of a minimal size data chunk, wherein X is known by the server and by at least a second client machine used for uploading a second file to the server in data chunks of size k'X; and uploading the first file to the server, wherein a first unique signature is calculated for the first file based on applying a signature function to a collection of signatures calculated for the minimal size data chunks of size X that make up the data chunks of size kX in the first file, wherein the uploading of the first file is accomplished by uploading the data chunks of size kX to the server in any order.

Efficient data deduplication in a data storage network

Systems and methods for estimating data reduction ratio for a data set is provided. The method comprises selecting a plurality of m elements from a data set comprising a plurality of N elements; associating an identifier hfor each of the plurality of m elements; associating an identifier hfor each of the plurality of elements in the data set; tracking number of times an element i appears in a base set that includes the plurality of m elements selected from the data set; calculating a value countthat indicates the number of times an identifier hmatches an identifier h; and estimating data reduction ratio for the plurality of N elements in the data set, based on number of m number elements selected from the data set and the value count.

Growing community of inventors

Haifa, Israel

Gil Vernik