Microsoft Technology Licensing, LLC

Thomas Hargrove

Mosha Pasumansky

Alexander Berger

Rajeev Prasad

Vladimir Schipunov

Systems and methods for generating distributed dataflow graphs and processing data elements in parallel utilizing the distributed dataflow graphs are provided. A sequential dataflow graph is formed from incoming data elements, and a variety of heuristics is applied to the sequential dataflow graph to determine which of the data transformation steps within the graph are capable of being processed multiple times in parallel. Once determined, the sequential dataflow graph is divided into subgraphs, which are then replicated, e.g., based on available resources and.or external constraints. The resulting subgraphs are connected, based on the semantics of each vertex, and a distributed dataflow graph is generated, which can efficiently process data elements, for instance, for data warehousing and the like.

Generating distributed dataflow graphs

A system, method, and related techniques are disclosed for processing data across a distributed network to a plurality of machines. The method may include receiving a first user-supplied transform and generating a first package based on the first user-supplied transform. The method may further include receiving a designated key and generating a second package based on the key. Furthermore, the method may include receiving a second user-supplied transform and generating a third package based on the second user-supplied transform. Moreover, the method may include distributing the first, second, and third packages to a plurality of machines using a cluster API.

Processing data across a distributed network

A system that facilitates data processing includes a receiver component that receives an SQL query. A partitioning component partitions the SQL query into multiple tasks and provides the tasks to multiple cluster nodes for processing. The system enables very large amounts of data (e.g., multiple terabytes) to be quickly prepared for analytical processing, such as for use in connection with a search engine, an advertisement provision system, etc.

Growing community of inventors

Kirkland, WA, United States of America

Thomas Hargrove