Emc IP Holding Company LLC

Emc Corporation

Lirong Jian

Tao Ma

Lei Chang

Zhanwei Wang

Lili Ma

Luke Lonergan

Gavin Sherry

Kurt Harriman

A method, article of manufacture, and apparatus for managing data. In some embodiments, this includes an initial instruction for a file stored in a first storage system, determining that the initial instruction is not supported by the first storage system, identifying a combination of instructions to the first storage system after determining that the initial instruction is not supported by the first storage system, wherein the combination of instructions is based on the initial instruction, performing the identified combination of instructions on the file stored in the first storage system, and storing results of the performed identified combination of instructions.

Pluggable storage system for distributed file systems

A query is executed in a massively parallel processing data storage system comprising a master node communicating with a cluster of multiple segments that access data in distributed storage by producing a self-described query plan at the master node that incorporates changeable metadata and information needed to execute the self-described query plan on the segments, and that incorporates references to obtain static metadata and information for functions and operators of the query plan from metadata stores on the segments. The distributed storage may be the Hadoop distributed file system, and the query plan may be a full function SQL query plan.

Self-described query execution in a massively parallel SQL execution engine

Data location-based query optimization for scan operators in a large-scale distributed database system is disclosed. In various embodiments, a set of query processing segments to be used to process a query is identified. It is determined for each of a plurality of data blocks to be scanned to process the query a corresponding storage location in which the data block is stored in a distributed storage system. Each of the plurality of data blocks is assigned to be scanned by a query processing segment selected based at least in part on a location of the query processing segment relative to the corresponding storage location in which the data block is stored.

Data locality based query optimization for scan operators

M&#xd7;N dispatching in a large scale distributed system is disclosed. In various embodiments, a query is received. A query plan is generated to perform the query. A subset of query processing segments is selected, from a set of available query processing segments, to perform an assigned portion of the query plan. An assignment to perform the assigned portion of the query plan is dispatched to the selected subset of query processing segments.

MxN dispatching in large scale distributed system

A system for sending a command stream comprises a first input interface, a processor, and an output interface. The first input interface is configured to receive a call. The processor is configured to determine one or more packets for the call. The one or more packets are able to be reassembled to form the call. The processor is configured to assign a first packet of the one or more packets to a first output queue and assign the second packet to a second output queue in the event there is a second packet of the one or more packets for the call. The output interface configured to send the first packet from the first output queue.

Supporting millions of parallel light weight data streams in a distributed system

A communication technique which includes determining, at least in part by comparing data associated with a packet that has been pulled from a received packet queue with a highest sequence number among packets that have been placed in the received packet queue, that the received packet queue has space available to receive a further packet. A receiver with which the received packet queue is associated is sent, based at least in part on the determination, a next packet.

Interconnect flow control

M X N dispatching in large scale distributed system

A transaction model for data stores using a distributed file system is disclosed. In various embodiments, a transaction-related metadata is sent to each of a plurality of segments to which a corresponding assigned portion of a query plan is assigned. A metadata modification record is received from zero or more of the segments, reflecting changes, if any, to system metadata required to be made to reflect changes to data made by that segment in processing the portion of the query plan assigned to that segment. It is determined whether a metadata modification record has been received from each of the segments in the plurality of segments. In the event a metadata modification record is determined to have been received from each of the segments in the plurality of segments, a system metadata store is updated to reflect the respective metadata modification records received from the plurality of segments.

Growing community of inventors

Beijing, China

Lirong Jian