Emc IP Holding Company LLC

Pivotal Software, Inc.

Emc Corporation

Milind Arun Bhandarkar

Sameer Tiwari

Bhooshan Deepak Mogal

Yuquan Lu

Wangda Tan

Jimmy Haijun Cao

David Stephen Reiner

David I Dietrich

A method, article of manufacture, and apparatus for managing data. In some embodiments, this includes determining a usage level of a file, wherein the file is stored in a first storage system, moving the file to a second storage system based on the determined usage level of the file updating location information in a catalog based on the movement of the file, and performing at least a portion of a query on the file after updating location information in the catalog.

Tiering with pluggable storage system for parallel query engines

A system for sharing a metadata store between a relational database and an unstructured data source is disclosed. The unstructured data source may comprise a Hadoop system with a Hadoop Distributed Files System.

Method for connecting a relational data store's meta data with hadoop

A method, article of manufacture, and apparatus for managing data. In some embodiments, this includes, receiving a query from a client, based on the received query, analyzing a catalog for location information, based on the analysis, determining a first storage system, an associated first file system and an associated first protocol, using the associated first protocol to communicate with the first storage system, and performing at least a portion of the query on the first storage system.

Pluggable storage system for parallel query engines

A method, article of manufacture, and apparatus for managing data. In some embodiments, this includes receiving a query from a client, based on the received query, analyzing a catalog for location information, based on the analysis, determining a first storage system, an associated first file system, an associated first protocol translator, a second storage system, an associated second file system, and an associated second protocol translator, identifying a first data and a second data, wherein the first data is stored on the first storage system, and the second data is stored on the second storage system, running a first job on the first data using the associated first protocol translator, wherein the first job is not a native job of the first file system, and running a second job on the second data using the associated second protocol translator, wherein the second job is not a native job of the second file system.

Pluggable storage system for parallel query engines across non-native file systems

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for a task centric resource scheduling framework. A scheduler executing on a master node computer of a distributed parallel computing system allocates computing resources of the parallel computing system to a program according to one or more policies associated with the program. Each policy includes a set of pre-determined computing resource constraints. Allocation of the computing resources includes performing multiple iterations of negotiation between the scheduler and a resource manager of the parallel computing system. In each iteration, a policy engine of the scheduler submits requests to get more resources from, or requests to release already acquired resources to, the resource manager. The policy engine generates the requests by balancing suggestions provided by analyzer components of the policy engine and a corresponding policy. The policy engine can then determine an allocation plan on how to allocate resources.

Iterative task centric resource scheduling for a user program between different computing frameworks

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for a distributed parallel computing system to adapt a foreign program to execute on the distributed parallel computing system. The foreign program is a program written for a computing framework that is different from a computing framework of the parallel computing system. The distributed parallel computing system includes a master node computer and one or more worker node computers. A scheduler executing on the master node computer acts as an intermediary between the foreign program and the parallel computing system. The scheduler negotiates with a resource manager of the parallel computing system to acquire computing resources. The scheduler then allocates the computing resources to the worker node computers as containers. The foreign program executes in the containers on the worker node computers in parallel.

Executing a foreign program on a parallel computing system

Method for connecting a relational data store's meta data with Hadoop

Classification of images or other types of high-resolution data is performed by a cluster-based data classification system. The system comprises a learner module, a classification director, and a complex classifier comprising a plurality of multi-outcome data classifiers. The classification director determines particular process rules and settings to be applied to a classification request, and the complex classifier is instantiated to process the classification request in accordance with the process rules and settings determined by the classification director. The process rules and settings are adapted under control of the learner module at least in part based on results obtained in processing the classification request and one or more additional classification requests. The cluster-based data classification system may be implemented on a processing platform comprising at least one Hadoop cluster, such that the classification is performed in a parallel manner across multiple processing devices utilizing MapReduce processing.

Growing community of inventors

San Jose, CA, United States of America

Milind Arun Bhandarkar