Emc Corporation

Pivotal Software, Inc.

Qubole, Inc.

Sivaramakrishnan Narayanan

Florian Michael Waas

Joy Jie Kent

Venkatesh Raghavan

Mohamed A Soliman

Lyublena Rosenova Antova

Chad Whipkey

Konstantinos Krikellas

Brian C Hagenbuch

Shrikanth Shankar

Joydeep Sen Sarma

Carlos Garcia-Alvarado

Mayank Ahuja

Chad William Whipkey

Systems and methods for automatically scaling a big data system. Methods include determining, at a first time, a first number of nodes for a cluster to process a request; assigning an amount of nodes equal to the first number of nodes to the cluster; determining a rate of progress of the request; determining, at a second time based on the rate of progress a second number of nodes; and modifying the amount of nodes to equal the second number of nodes. Systems include a cluster manager, to add and/or remove any nodes; the big data system, to process requests that utilize the cluster and nodes, and an automatic scaling cluster manager including a big data interface for communicating with the big data system; a cluster manager interface for communicating with the cluster manager; and a cluster state machine.

Systems and methods for auto-scaling a big data system

SQL queries are optimized to operate directly on compressed data (and obtain the correct result) rather than requiring that the data be first decompressed prior to processing a query. Certain characteristic pattern trees are mapped against a logical input query plan that includes certain logical operators such as a DECOMPRESS that precedes a JOIN or a GROUPBY in association with a COUNT to identify instances in the plan that match a characteristic pattern. Upon locating a match, the input query plan is transformed into a logically equivalent plan that operates correctly on compressed data, by analyzing the interplay of the semantics of logical query operations with the compressed data and substituting less costly structures and operations. DECOMPRESS operations are moved to operate subsequent to a JOIN or eliminated altogether, and COUNT operations are replaced by a different operation, such as SUM, that is logically equivalent for compressed data.

RLE-aware optimization of SQL queries

A request for a metadata object is received where the request includes an identifier. It is determined whether the metadata object is stored in a metadata cache, including by: selecting a sub-cache in the metadata cache based at least in part on the identifier and determining whether the metadata object is stored in that sub-cache. If it is determined the metadata object is not in the metadata cache, a request (in a format associated with the source) to obtain the metadata object is sent to a source with which the metadata object is associated. If the metadata object provided by the source is in a native format which is not the same as the common format, the metadata object is transformed from the native format into the common format. Access to the metadata object in the common format is provided.

Metadata cache supporting multiple heterogeneous systems

A method, article of manufacture, and apparatus for processing queries, comprising analyzing a query tree, determining at least one operator based on the query tree analysis, assigning a memory allocation for each of the at least one operator, and storing the assignment in a storage device. In some embodiments, a memory classification for each of the at least one operator is determined. In some embodiments, assigning a memory allocation for each of the at least one operator includes assigning a memory allocation based on the memory classification.

Memory quota

A method, article of manufacture, and apparatus for processing queries, comprising receiving a query, determining a query classification for the query, assigning the query to a resource queue based on the determined query classification, and placing the query in the assigned resource queue. In some embodiments, the resource queue is divided into a plurality of slots, and the query is placed in a slot. The resource queue may be associated with a resource queue memory allocation, and each of the plurality of slots is associated with a slot memory allocation.

Resource queues

A method of analyzing the performance of a query optimizer includes identifying an event trigger. A reproduction object characterizing operational parameters of the customer computer at the time of the event trigger is populated. The reproduction object is transported from the customer computer to a test computer. The reproduction object is analyzed at the test computer to characterize the performance of a query optimizer.

Apparatus and method for analyzing query optimizer performance

A method, article of manufacture, and apparatus for processing queries, comprising performing a query with a first memory intensive allocation, receiving an indication that a memory intensive operator spilled to a first storage device, adjusting a resource queue memory allocation based on the indication, computing a second memory intensive allocation based on the adjusted resource queue memory allocation, and storing the second memory intensive allocation in a second storage device.

Dynamic allocation of memory for memory intensive operators

Data movement driven placement of data in a distributed database is disclosed. A set of one or more queries anticipated to be performed with respect to a data set is received. A set of statistics associated with the data set is received. The set of queries and the statistics are used to explore algorithmically a search space of distribution policy alternatives, to determine an optimal distribution policy to distribute data comprising the data set across a plurality of database servers.

Data movement driven automatic data placement

The invention enables a correlated or multi-row subquery (CSQ) to be performed on distributed MPP and shared-nothing databases by broadcasting intermediate results, prior to a correlation operation, from subquery execution on one segment to all other segments in the distributed database so that the respective CSQs of each segment will have access to the necessary results to permit correct execution of the CSQ. Additionally, the intermediate results are saved to disk to avoid the necessity of replicating the same intermediate results multiple times during execution of a subquery plan.

Executing correlated and multi-row subqueries in a MPP database

A request for a metadata object is received. It is determined whether the metadata object is stored in a metadata cache. In the event the metadata object is not stored in the metadata cache, a request to obtain the metadata object is sent to a source with which the metadata object is associated in a format associated with the source. The metadata object is transformed from a native format of the source into a common format if the metadata object as provided by the source is in a native format that is not the same as the common format. Access to the metadata object in the common format is provided.

A shared nothing distributed database system includes a master node and a set of shared nothing nodes. Each shared nothing node includes a node state table stored in memory that characterizes various statements and multiple processes implementing each statement, target resource consumption rate for each process and process activity information. A monitor module executed by a central processing unit processes information in the node state table and adjusts values in the node state table in accordance with priority criteria specifying individual user priority ascribed to a statement. A query associated with a statement is processed in the shared nothing distributed database system in accordance with the priority criteria.

Growing community of inventors

Sunnyvale, CA, United States of America

Sivaramakrishnan Narayanan