Hewlett-packard Development Company, L.p.

Google Inc.

Hewlett Packard Enterprise Development LP

Xiaozhou Li

Eric A Anderson

Mehul A Shah

John Johnson Wylie

Kenneth Jerome Goldman

Jelena Pjesivac-Grbovic

Sunghwan Ihm

Christopher Edward Hoover

Joseph Andrew Tucek

Charles E Christian, Jr

Tim Reddin

Robert J Souza

Ludmila Cherkasova

Mustafa Uysal

Xin Zhang

Yonggang Zhao

Marian Dvorsky

Wojciech Golab

Amitanand Aiyer

Allstair Veitch

Ovidiu Gheorghioiu

Kenneth J Goldman

A manager agent access a grain hierarchy that represents a state for a computation that is to performed by a distributed processing system, wherein the grain hierarchy includes manager agent grains including metadata for manager agent processes that manage the performance of the computation by the distributed processing system, and worker agent grains including, for tasks to be performed by the distributed processing system, metadata for worker agents that each correspond to a subset of the plurality of data processors for performing the task. A manager agent performs processes defined by a manager agent grain to manage the computation by worker agents and storing, within the grain, metadata describing the manager agent process performed by the manager agent, and worker agents perform tasks assigned to the worker agents based on an assignment of a respective worker agent grain to the worker agent.

Managing metadata for a distributed processing system with manager agents and worker agents

Managing metadata hierarch for a distributed processing system with depth-limited hierarchy subscription

Methods, systems, and apparatus are described for managing metadata for a distributed processing system. In one aspect, a method includes receiving a computation specification that includes a set of grains that specifies an initial state for a computation that is to be performed by a distributed processing system, wherein each grain comprises metadata that specifies a portion of the initial state for the computation; storing a grain hierarchy that represents a state for the computation based on a grain type associated with each grain, the grain hierarchy comprising subscription grains for subscriptions for the grain hierarchy, each subscription corresponding to one or more grains included in the grain hierarchy, and each subscription specifying one or more actions to be performed by the hub device; and performing, during performance of the computation, at least one of the actions specified by at least one of the subscriptions.

A method is provided for scrubbing information stored in a data storage system where the information is stored as a plurality of encoded fragments across multiple storage devices. The method includes maintaining on a first storage device a list of metadata entries corresponding to values that are stored in the data storage system at an At Maximum Redundancy (AMR) state, verifying that encoded fragments associated with each of the metadata entries are stored on a second storage, verifying that a corresponding metadata entry is stored on the first storage device for each encoded fragment that is stored on the second storage device, and scheduling for recovery any missing encoded fragments and/or any missing metadata entry.

Scrubbing procedure for a data storage system

A method for shard assignment in a large-scale data processing job is provided. Datasets are divided into a plurality of shards and the shards are indexed and aggregated into one or more groups. A worker process is initially assigned an indexed shard from a group. The initial assignment can assigned based on a simple algorithm. The worker's subsequent shard assignment is based on the index of the initially assigned shard.

Method for shard assignment in a large-scale data processing job

Embodiments include methods, apparatus, and systems for retrieving data from multiple locations in storage systems. One embodiment includes a method that determines that data is stored in multiple locations remote to a computer, estimates a latency to retrieve the data from the multiple locations, and requests the data from the plural locations.

Retrieving data from multiple locations in storage systems

A system comprises a processor and storage containing software executable by the processor. The storage also contains a trace log that contains information pertaining to read and write operations and, for each read and write operation, the information is indicative of a start time, a completion time, and a value targeted by the read or write operation, Based on the trace log, the software causes the processor to construct a directed graph comprising nodes as well as edges interconnecting at least some of the nodes, each node representing a read or write operation and determine whether the constructed directed graph has a cycle. At least one edge is at least one of a data edge representing a data precedence between operations and a time edge representing a time precedence between operations, and at least one edge is a hybrid edge representing both time and data precedence between operations.

Directed graphs pertaining to read/write operations

A method of recovering a value in a data storage system includes maintaining a list of entries corresponding to values that have been inserted into the data storage system, determining whether the values are at an At Maximum Redundancy (AMR) state, and if not, the converging the values to the AMR state.

Recovery procedure for a data storage system

A method for verifying a consistency level in a key-value store, in which a value is stored in a cloud-based storage system comprising a read/write register identified by a key. At a centralized monitor node, a history of operations including writes and reads performed at the key is created, and a distance between a read of a value at the key and a latest write to the key is determined. It can then be ascertained whether the distance satisfies a relaxed atomicity property.

Verifying consistency levels

Systems and methods of enhanced backup job scheduling are disclosed. An example method may include determining a number of jobs (n) in a backup set, determining a number of tape drives (m) in the backup device, and determining a number of concurrent disk agents (maxDA) configured for each tape drive. The method may also include defining a scheduling problem based on n, m, and maxDA. The method may also include solving the scheduling problem using an integer programming (IP) formulation to derive a bin-packing schedule that minimizes makespan (S) for the backup set.

Enhanced backup job scheduling

A method is provided for efficiently recovering information in a distributed storage system where a list of values that should be stored on a storage device is maintained. A first convergence round is scheduled to be performed on the list of values to bring each value to an At Maximum Redundancy (AMR) state. A second convergence round is scheduled to be performed on the list by selecting a wait time interval from a predefined range of wait time intervals between starts of convergence rounds.

Efficient failure recovery in a distributed data storage system

A server receives information that identifies which chunks are stored in local caches at client computers and receives a request to evict a chunk from a local cache of a first one of the client computers. The server determines whether the chunk stored at the local cache of the first one of the client computers is globally oldest among the chunks stored in the local caches at the client computers, and authorizes the first one of the client computers to evict the chunk when the chunk is the globally oldest among the chunks stored in the local caches at the client computers.

Computer system with cooperative cache

Erasure-encoded data is stored across a plurality of storage devices in a data storage system. The erasure-encoded data includes k data elements to store on k data storage devices and m parity elements to store on m parity storage devices, wherein for a given minimum Hamming distance d of the data storage system and m&#x2267;(d&#x2212;1), data elements are assigned only to corresponding unique combinations of parity elements of size (d&#x2212;1).

Non-MDS erasure codes for storage systems

A method for measuring consistability of a distributed storage system is disclosed. The method includes determining at least one consistency level that the distributed storage system can provide. A plurality of failure classes can be determined for the distributed storage system. A probability of the distributed storage system to be in each of the plurality of failure classes can be measured. Each failure class can be mapped to the at least one consistency level. The probability of each failure class for each consistency level can be summed to determine an expected portion of time that the distributed storage system provides each consistency level.

Growing community of inventors

Cupertino, CA, United States of America

Xiaozhou Li