Google Inc.

The Trustees of Princeton

Sunghwan Ihm

Jelena Pjesivac-Grbovic

Xiaozhou Li

Kenneth Jerome Goldman

Craig D Chambers

Kym Hines

Mariana Sharp

Danijela Mijailovic

Ben Walters

Anirudh Badam

KyoungSoo Park

Vivek Pai

Larry L Peterson

Kenneth J Goldman

Aspects of the disclosure relate to evaluating the rollout of a change to a data processing framework (DPF). A first set of jobs is selected out of a plurality of jobs. The first set of jobs is processed by the DPF with the proposed change enabled and the other jobs of the plurality of jobs by the DPF without the change enabled. A performance cost value is determined for each of the jobs in the first set of jobs processed by the DPF with the proposed change enabled and for each of the other jobs of the plurality of jobs processed by the DPF with the proposed change disabled. Based on the performance cost values of the jobs in the first set of jobs and the performance cost values of the other jobs of the plurality of jobs, a determination is made whether the proposed change is beneficial to the DPF.

System for automatically evaluating a change in a large population of processing jobs

A manager agent access a grain hierarchy that represents a state for a computation that is to performed by a distributed processing system, wherein the grain hierarchy includes manager agent grains including metadata for manager agent processes that manage the performance of the computation by the distributed processing system, and worker agent grains including, for tasks to be performed by the distributed processing system, metadata for worker agents that each correspond to a subset of the plurality of data processors for performing the task. A manager agent performs processes defined by a manager agent grain to manage the computation by worker agents and storing, within the grain, metadata describing the manager agent process performed by the manager agent, and worker agents perform tasks assigned to the worker agents based on an assignment of a respective worker agent grain to the worker agent.

Managing metadata for a distributed processing system with manager agents and worker agents

Managing metadata hierarch for a distributed processing system with depth-limited hierarchy subscription

Methods, systems, and apparatus are described for managing metadata for a distributed processing system. In one aspect, a method includes receiving a computation specification that includes a set of grains that specifies an initial state for a computation that is to be performed by a distributed processing system, wherein each grain comprises metadata that specifies a portion of the initial state for the computation; storing a grain hierarchy that represents a state for the computation based on a grain type associated with each grain, the grain hierarchy comprising subscription grains for subscriptions for the grain hierarchy, each subscription corresponding to one or more grains included in the grain hierarchy, and each subscription specifying one or more actions to be performed by the hub device; and performing, during performance of the computation, at least one of the actions specified by at least one of the subscriptions.

The illustrative embodiments described herein are directed to a system, methods, and computer program product, for network acceleration and efficient indexing for caching file systems. For example, the disclosed embodiments achieve (1) high compression rate, (2) low disk seeks, and (3) low memory pressure using by introducing novel features such as, but not limited to, multi-resolution chunking and intelligent load shedding. In addition, the disclosed embodiments introduces a family of HashCache policies that enable large size web caching with little or no use of main memory.

Systems and methods for network acceleration and efficient indexing for caching file systems

Growing community of inventors

Sunnyvale, CA, United States of America

Sunghwan Ihm