Google Inc.

Kenneth Jerome Goldman

Jelena Pjesivac-Grbovic

Yonggang Zhao

Xiaozhou Li

Sunghwan Ihm

Stephen P G Gildea

Chi Ma

Tushar Deepak Chandra

Tal Shaked

Jerry Zhao

Jeremy S Hurwitz

Eugene R Kirpichov

Anthony Mancuso

Matthew Faulkner

Wesley Kendall

A method includes receiving, by a data processing apparatus, a plurality of file operation requests, each file operation request including a priority, a deadline, and an operation type and representing a request to perform an operation on at least one file maintained in a distributed file system; identifying, by the data processing apparatus, a group of file operation requests to be executed together from the plurality of file operation requests, the identification based at least in part on at least one of: the file operations in the group of file operations being directed to a same storage system, or file operations in the group of file operations sharing a common operation type; and sending a request to execute the group of file operation requests to a system configured to perform the group of file operation requests.

File operation task optimization

A manager agent access a grain hierarchy that represents a state for a computation that is to performed by a distributed processing system, wherein the grain hierarchy includes manager agent grains including metadata for manager agent processes that manage the performance of the computation by the distributed processing system, and worker agent grains including, for tasks to be performed by the distributed processing system, metadata for worker agents that each correspond to a subset of the plurality of data processors for performing the task. A manager agent performs processes defined by a manager agent grain to manage the computation by worker agents and storing, within the grain, metadata describing the manager agent process performed by the manager agent, and worker agents perform tasks assigned to the worker agents based on an assignment of a respective worker agent grain to the worker agent.

Managing metadata for a distributed processing system with manager agents and worker agents

Managing metadata hierarch for a distributed processing system with depth-limited hierarchy subscription

The present disclosure relates to dynamically adjusting shard allocation during parallel processing operations. One example method includes determining a target completion time for a batch data processing job of an input data set performed by a plurality of tasks, each of the plurality of tasks processing a different input shard including a different portion of the input data set; identifying a first task having an estimated completion time greater than the target completion time of the batch data processing job; and splitting the first input shard into a first split input shard and a second split input shard different from the first split input shard, the first split input shard including a first portion of the first input shard, and the second split input shard including a second portion of the first input shard different from the first portion.

Dynamic shard allocation adjustment

Parallel processing of data may include a set of map processes and a set of reduce processes. Each map process may include at least one map thread. Map threads may access distinct input data blocks assigned to the map process, and may apply an application specific map operation to the input data blocks to produce key-value pairs. Each map process may include a multiblock combiner configured to apply a combining operation to values associated with common keys in the key-value pairs to produce combined values, and to output intermediate data including pairs of keys and combined values. Each reduce process may be configured to access the intermediate data output by the multiblock combiners. For each key, an application specific reduce operation may be applied to the combined values associated with the key to produce output data.

Parallel processing of data

A method and a system for variable aggregation among large-scale data processing workers is discussed. Workers join groups and add variables to the groups. One worker is elected as the group leader. Workers post values to group variables, which are aggregated by the leader. The leader also announces the aggregated values. This variable aggregation allows for asynchronous communication among data processing workers.

System and method for variable aggregation in order for workers in a data processing to share information

The present disclosure relates to cataloging data sets for reuse in pipeline applications. One example method includes identifying a data set produced by a particular pipeline object included in a first pipeline instance, the first pipeline instance including a plurality of pipeline objects, each pipeline object configured to perform a computation, and the particular pipeline object configured to perform a particular computation; determining a set of metadata for the data set, the set of metadata including identifying information for the data set to identify the data set to pipeline instances separate from the first pipeline instance; and allowing pipeline instances separate from the first pipeline instance to retrieve the data set based at least in part on the set of metadata, wherein the pipeline instances avoid performing the particular computation by using the retrieved data set.

Cataloging data sets for reuse in pipeline applications

Methods, systems, and apparatus are described for managing metadata for a distributed processing system. In one aspect, a method includes receiving a computation specification that includes a set of grains that specifies an initial state for a computation that is to be performed by a distributed processing system, wherein each grain comprises metadata that specifies a portion of the initial state for the computation; storing a grain hierarchy that represents a state for the computation based on a grain type associated with each grain, the grain hierarchy comprising subscription grains for subscriptions for the grain hierarchy, each subscription corresponding to one or more grains included in the grain hierarchy, and each subscription specifying one or more actions to be performed by the hub device; and performing, during performance of the computation, at least one of the actions specified by at least one of the subscriptions.

Aspects of the disclosure relate to learning the most useful backup tasks when processing a job in a distributed computing application. For example, training data may be generated by running a job and running backup tasks for some of the tasks for that job. List of features may be generated for each backup task at different times. A representative list of features may be labeled according to the backup tasks usefulness to the job. The lists and usefulness labels may be used to generate a model of usefulness. The model may then be used to evaluate the usefulness of the tasks for the next job. This information may then be used to schedule backup tasks for execution. In addition, lists of features for these backup tasks may be generated and then used to update the model in a continuous learning loop.

Growing community of inventors

Palo Alto, CA, United States of America

Kenneth Jerome Goldman