Emc IP Holding Company LLC

Ofri Masad

Patricia Gomes Soares Florissi

Ido Singer

Sudhir Vijendra

Alon Grubshtein

Raphael Cohen

An apparatus comprises a processing platform configured to implement a machine learning system for automated classification of documents comprising text data of at least one database. The machine learning system comprises a clustering module configured to assign each of the documents to one or more of a plurality of clusters corresponding to respective topics identified from the text data in accordance with at least one topic model, and an interface configured to present portions of documents assigned to a particular one of the clusters by the clustering module and to receive feedback regarding applicability of the corresponding topic to each of one or more of the presented portions on a per-portion basis. The topic model is updated based at least in part on the received feedback. The feedback may comprise, for example, selection of a confidence level for applicability of the topic to a given one of the presented portions.

Topic model based clustering of text data with machine learning utilizing interface feedback

An apparatus in one embodiment comprises at least one processing device having a processor coupled to a memory. The processing device is configured to initiate distributed computations across a plurality of data processing clusters associated with respective data zones, and to combine local processing results of the distributed computations from respective ones of the data processing clusters. Each of the data processing clusters is configured to process data from a data source of the corresponding data zone using a local data structure and an associated computational framework of that data processing cluster. A first one of data processing clusters utilizes a first local data structure configured to support a first computational framework, and at least a second one of the data processing clusters utilizes a second local data structure different than the first local data structure and configured to support a second computational framework different than the first computational framework.

Scalable distributed computations utilizing multiple distinct computational frameworks

An apparatus in one embodiment comprises at least one processing device having a processor coupled to a memory. The processing device is configured to distribute in-memory computations across at least first and second nodes of respective distinct data processing clusters of a plurality of data processing clusters over at least one network, and to aggregate results of the distributed in-memory computations for delivery to a requesting client device. The data processing clusters are associated with respective distinct data zones, and the first and second nodes of the respective distinct data processing clusters are configured to perform corresponding portions of the distributed in-memory computations utilizing respective ones of first and second in-memory datasets locally accessible within their respective data zones. The in-memory computations in some embodiments illustratively comprise Spark computations, such as Spark Core batch computations. The in-memory datasets in such an arrangement may comprise respective Spark resilient distributed datasets.

Scalable distributed in-memory computation

An analytics platform is configured to receive a request to execute a distributed analytics workflow specification comprising multiple distributed computations to be performed across a plurality of data processing clusters associated with respective data zones. For each of a plurality of analytic services to be executed in accordance with the distributed analytics workflow specification, the platform determines if execution of the analytic service is authorized for the request, and for each of a plurality of datasets to be operated on by one or more of the analytic services in accordance with the distributed analytics workflow specification, the platform determines if execution of the one or more analytic services on the dataset is authorized by its corresponding data zone for the request. Responsive to each of the analytic services and each of the datasets being authorized for the request, the platform initiates execution of the distributed computations of the distributed analytics workflow specification across the data processing clusters.

Analytics platform for scalable distributed computations

An apparatus is configured to initiate distributed computations across a plurality of data processing clusters associated with respective data zones, to utilize local processing results of at least a subset of the distributed computations from respective ones of the data processing clusters to generate global processing results, and to update at least one distributed ledger maintained by one or more of the plurality of data processing clusters to incorporate one or more blocks each characterizing at least a portion of the distributed computations. Each of at least a subset of the data processing clusters is configured to process data from a data source of the corresponding data zone using one or more local computations of that data processing cluster to generate at least a portion of the local processing results. At least one of the data processing clusters is configured to apply one or more global computations to one or more of the local processing results to generate at least a portion of the global processing results.

Blockchain integration for scalable distributed computations

An apparatus in one embodiment comprises at least one processing device having a processor coupled to a memory. The processing device is configured to receive results of intermediate statistical computations performed on respective ones of a plurality of datasets in respective ones of a plurality of distributed processing nodes configured to communicate over at least one network. The processing device is further configured to perform at least one global statistical computation based at least in part on the results of the intermediate statistical computations, and to utilize a result of the global statistical computation to perform one or more benchmarking operations for specified parameters relating to the plurality of datasets. The distributed processing nodes are associated with respective distinct data zones in which the respective datasets are locally accessible to the respective distributed processing nodes. At least a subset of the receiving, performing and utilizing are repeated in each of a plurality of iterations.

Global benchmarking and statistical analysis at scale

A computing device of a data zone includes a persistent storage and a processor. The persistent storage includes a locked data batch of the data zone. The processor obtains an upstream computation request; instantiates a computation framework to process the locked data batch based on a global data batch specified in the upstream computation request; instantiates a downstream computation manager to manage a downstream computation; and instantiates, by the downstream computation manager, a second computation framework in a second computing device of a second data zone to process a second locked data batch of the second data zone.

Multi-framework distributed computation

An analytics platform is configured to provide at least one interface permitting registration of datasets of respective data zones for use in performing distributed computations across a plurality of data processing clusters associated with the respective data zones. The analytics platform is further configured to maintain one or more analytic service libraries each comprising a plurality of analytic services for use in performing the distributed computations, and to initiate execution of a particular subset of the distributed computations, responsive to a corresponding request, using selected ones of the registered datasets and selected ones of the analytic services. The analytics platform allocates to an entity associated with one of more of the selected registered datasets one or more specified credits responsive to successful completion of the particular subset of the distributed computations. The analytic services are illustratively executed on the datasets without the datasets leaving their respective data zones.

An apparatus in one embodiment comprises at least one processing device having a processor coupled to a memory. The processing device is configured to initiate distributed data streaming computations across data processing clusters associated with respective data zones, and in each of the data processing clusters, to separate a data stream provided by a data source of the corresponding data zone into a plurality of data batches and process the data batches to generate respective result batches. Multiple ones of the data batches across the data processing clusters are associated with a global data batch data structure, and multiple ones of the result batches across the data processing clusters are associated with a global result batch data structure based at least in part on the global data batch data structure. The result batches are processed in accordance with the global result batch data structure to generate one or more global result streams.

Scalable distributed data streaming computations across multiple data processing clusters

An apparatus in one embodiment comprises at least one processing device having a processor coupled to a memory. The processing device is configured to initiate distributed computations across a plurality of data processing clusters associated with respective data zones, and to combine local processing results of the distributed computations from respective ones of the data processing clusters. The data processing clusters are configured to perform respective portions of the distributed computations by processing data local to their respective data zones utilizing at least one local data structure configured to support at least one computational framework. A first one of data processing clusters is implemented in a first cloud of a first type provided by a first cloud service provider. At least a second one of the data processing clusters is implemented in a second cloud of a second type different than the first type, provided by a second cloud service provider.

Scalable distributed computations utilizing multiple distinct clouds

An apparatus in one embodiment comprises at least one processing device having a processor coupled to a memory. The processing device is configured to distribute in-memory computations across a plurality of data processing clusters associated with respective data zones, and to combine local processing results of the distributed in-memory computations from the data processing clusters. The distributed in-memory computations utilize local data structures of respective ones of the data processing clusters. A given one of the local data structures in one of the data processing clusters receives local data of the corresponding data zone and is utilized to generate the local processing results of that data processing cluster that are combined with local processing results of other ones of the data processing clusters. The local data structures are configured to support batch mode extensions such as Spark SQL, Spark MLlib or Spark GraphX for performance of the distributed in-memory computations.

Growing community of inventors

Beer-Sheva, Israel

Ofri Masad