Google Inc.

Samuel Green McVeety

Vyacheslav Alekseyevich Chernyak

Craig D Chambers

Daniel Mills

Robert Bradshaw

Reuven Lax

Manuel Alfred Fahndrich

Mark Brian Shields

Ahmet Altay

Andrei Maksimenka

Harris Samuel Nover

Rafael De Jesús Fernández Moctezuma

Mikhail Shmulyan

Cosmin Ionel Arad

Jeffrey Paul Gardner

Samuel Carl Whittle

Methods, systems, and apparatus are described for throttling a distributed processing system. In one aspect, a method includes identifying records being processed by a distributed processing system that performs agent processes, each of the records including a corresponding timestamp; determining, based on timestamps of the records that have been processed by a first agent process, a first agent progress; identifying a dependent agent process performed by the distributed processing system, wherein the dependent agent process processes only records that have been processed by the first agent process; determining, based on timestamps of records that have been processed by the dependent agent process, a dependent agent progress; and throttling performance of the first process based on the first agent progress and the dependent agent progress.

Distributed processing system throttling using a timestamp

A job that receives as input a data stream is executed. For the job, it is iteratively determined a backlog growth over a first period of time; a backlog volume; and whether to adjust a quantity of processing resources. For each iteration for which the quantity of processing resources allocated to the job are determined to be adjusted, adjusting the quantity of processing resources allocated to the job. For each iteration for which the quantity of processing resources allocated to the job are determined not to be adjusted, maintaining the quantity of processing resources allocated to the job.

Resource allocation for computer processing

Distributed processing system throttling

Growing community of inventors

Seattle, WA, United States of America

Samuel Green McVeety