Emc IP Holding Company LLC

Pivotal Software, Inc.

Luke Lonergan

Tao Ma

Lei Chang

Zhanwei Wang

Lirong Jian

Lili Ma

Jeffrey Ira Cohen

Alon Goldshuv

Caleb E Welton

John Glenn Eshleman

Matthew Clark McCline

Kurt Harriman

Dov Yaron Dorin

Alex Shacked

A query is executed in a massively parallel processing data storage system comprising a master node communicating with a cluster of multiple segments that access data in distributed storage by producing a self-described query plan at the master node that incorporates changeable metadata and information needed to execute the self-described query plan on the segments, and that incorporates references to obtain static metadata and information for functions and operators of the query plan from metadata stores on the segments. The distributed storage may be the Hadoop distributed file system, and the query plan may be a full function SQL query plan.

Self-described query execution in a massively parallel SQL execution engine

Data location-based query optimization for scan operators in a large-scale distributed database system is disclosed. In various embodiments, a set of query processing segments to be used to process a query is identified. It is determined for each of a plurality of data blocks to be scanned to process the query a corresponding storage location in which the data block is stored in a distributed storage system. Each of the plurality of data blocks is assigned to be scanned by a query processing segment selected based at least in part on a location of the query processing segment relative to the corresponding storage location in which the data block is stored.

Data locality based query optimization for scan operators

M&#xd7;N dispatching in a large scale distributed system is disclosed. In various embodiments, a query is received. A query plan is generated to perform the query. A subset of query processing segments is selected, from a set of available query processing segments, to perform an assigned portion of the query plan. An assignment to perform the assigned portion of the query plan is dispatched to the selected subset of query processing segments.

MxN dispatching in large scale distributed system

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for streaming external data in parallel from a second distributed system to a first distributed system. One of the methods includes receiving a query that requests a join of first rows of a first table in a first distributed system with second rows of an external table, the external table representing data in a second distributed system. Each of the segment nodes communicates with a respective extension service that obtains fragments from one or more data nodes of the second distributed system according to location information for the respective fragments, and provides to the segment node a stream of data corresponding to second rows of the external table. Each of the segment nodes computes joined rows between the first rows of the first table and the stream of data corresponding to second rows of the external table.

Parallel streaming of external data

A system for sending a command stream comprises a first input interface, a processor, and an output interface. The first input interface is configured to receive a call. The processor is configured to determine one or more packets for the call. The one or more packets are able to be reassembled to form the call. The processor is configured to assign a first packet of the one or more packets to a first output queue and assign the second packet to a second output queue in the event there is a second packet of the one or more packets for the call. The output interface configured to send the first packet from the first output queue.

Supporting millions of parallel light weight data streams in a distributed system

M X N dispatching in large scale distributed system

A transaction model for data stores using a distributed file system is disclosed. In various embodiments, a transaction-related metadata is sent to each of a plurality of segments to which a corresponding assigned portion of a query plan is assigned. A metadata modification record is received from zero or more of the segments, reflecting changes, if any, to system metadata required to be made to reflect changes to data made by that segment in processing the portion of the query plan assigned to that segment. It is determined whether a metadata modification record has been received from each of the segments in the plurality of segments. In the event a metadata modification record is determined to have been received from each of the segments in the plurality of segments, a system metadata store is updated to reflect the respective metadata modification records received from the plurality of segments.

Transaction model for data stores using distributed file systems

A computer readable storage medium includes executable instructions to define a map-reduce document that coordinates processing of data in a distributed database. The map-reduce document complies with a map-reduce specification that integrates map-reduce functions with queries in a query language. The operations specified by the map-reduce document are executed in the distributed database.

Integrating map-reduce into a distributed relational database

A computer readable storage medium includes executable instructions to store load data in new rows of an append only table within a data warehouse, without updating or deleting existing rows in the append only table. The new rows are added in a database transaction that guarantees that the distributed reads and writes of all transactions are processed reliably and that the transactions obey snapshot isolation rules. Data changes associated with the load data are maintained in a separate update table. Data from the append only table is merged with data changes from the update table to supply read data.

Growing community of inventors

San Carlos, CA, United States of America

Luke Lonergan