Google Inc.

University of Washington

Craig D Chambers

Frances J Perry

Ashish Raniwala

Robert Bradshaw

Robert R Henry

Stephen Robert Adams

Nathan Weizenbaum

Jordan Tigani

Daniel Mills

Reuven Lax

Ezra Cooper

Tyler Arthur Akidau

Stephen R Adams

Jelena Pjesivac-Grbovic

Sunghwan Ihm

Ben Chambers

Mariana Sharp

Kym Hines

Danijela Mijailovic

Ben Walters

Matthai Philipose

Manuel Alfred Fahndrich

Susan Jane Eggers

Samuel Green McVeety

Mark Brian Shields

Ahmet Altay

Rafael De Jesús Fernández Moctezuma

Andrei Maksimenka

Harris Samuel Nover

Mikhail Shmulyan

Samuel Carl Whittle

Cosmin Ionel Arad

Jeffrey Paul Gardner

Markus Mock

Brian K Grant

A data parallel pipeline may specify multiple parallel data objects that contain multiple elements and multiple parallel operations that operate on the parallel data objects. Based on the data parallel pipeline, a dataflow graph of deferred parallel data objects and deferred parallel operations corresponding to the data parallel pipeline may be generated and one or more graph transformations may be applied to the dataflow graph to generate a revised dataflow graph that includes one or more of the deferred parallel data objects and deferred, combined parallel data operations. The deferred, combined parallel operations may be executed to produce materialized parallel data objects corresponding to the deferred parallel data objects.

Parallel processing of data

Aspects of the disclosure relate to evaluating the rollout of a change to a data processing framework (DPF). A first set of jobs is selected out of a plurality of jobs. The first set of jobs is processed by the DPF with the proposed change enabled and the other jobs of the plurality of jobs by the DPF without the change enabled. A performance cost value is determined for each of the jobs in the first set of jobs processed by the DPF with the proposed change enabled and for each of the other jobs of the plurality of jobs processed by the DPF with the proposed change disabled. Based on the performance cost values of the jobs in the first set of jobs and the performance cost values of the other jobs of the plurality of jobs, a determination is made whether the proposed change is beneficial to the DPF.

System for automatically evaluating a change in a large population of processing jobs

A method includes receiving data corresponding one of streaming data or batch data and a content of the received data for computation. The method also includes determining an event time of the data for slicing the data, determining a processing time to output results of the received data, and emitting at least a portion of the results of the received data based on the processing time and the event time.

Data flow windowing and triggering

One example method includes identifying synchronous code including instructions specifying a computing operation to be performed on a set of data; transforming the synchronous code into a pipeline application including one or more pipeline objects; identifying a first input data set on which to execute the pipeline application; executing the pipeline application on a first input data set to produce a first output data set; after executing the pipeline application on the first input data set, identifying a second input data set on which to execute the pipeline application; determining a set of differences between the first input data set and second input data set; and executing the pipeline application on the set of differences to produce a second output data set.

Incremental parallel processing of data

A job that receives as input a data stream is executed. For the job, it is iteratively determined a backlog growth over a first period of time; a backlog volume; and whether to adjust a quantity of processing resources. For each iteration for which the quantity of processing resources allocated to the job are determined to be adjusted, adjusting the quantity of processing resources allocated to the job. For each iteration for which the quantity of processing resources allocated to the job are determined not to be adjusted, maintaining the quantity of processing resources allocated to the job.

Resource allocation for computer processing

An untrusted application is received at a data center including one or more processing modules and providing a native processing environment. The untrusted application includes a data parallel pipeline. Secured processing environments are used to execute the untrusted application.

Parallel processing of data for an untrusted application

Growing community of inventors

Seattle, WA, United States of America

Craig D Chambers