Google Inc.

Lucent Technologies Inc.

Sean Quinlan

Robert Cypher

Steven Robert Schirripa

Sean Matthew Dorward

Jeffrey Adgate Dean

Sanjay Ghemawat

Alexander Lloyd

Wilson Cheng-Yi Hsieh

Michael James Boyer Epstein

Christian Eric Schrock

Peter H Hochschild

Lawrence Elias Greenfield

Robert C Pike

Priyanka Gupta

Michael Burrows

Lidor Carmi

Joshua A Redstone

Andrew Fikes

Yasushi Saito

Sebastian Kanthak

Christopher Scott Taylor

Michal Piotr Szymaniak

Lavina Jain

Generally disclosed herein is an approach to migrate data from a first type of distributed system to a second type of distributed system without locking data, where transactional dual writes are not available across the two systems. The approach starts by setting up a bi-directional replication between the first system and the second system. The first system will initially operate as a primary system, where the primary system receives and serves write requests from clients or other devices. For each write to the first system, the second system is updated with an asynchronous write. When the second system is caught up to the first system, such that both the first and second systems reflect approximately the same data, the second system can be switched over to serve as the primary system. The second system can now directly receive and serve all future read and write requests.

Online migration from an eventually consistent system to a strongly consistent system

The present technology proposes techniques for ensuring globally consistent transactions. This technology may allow distributed systems to ensure the causal order of read and write transactions across different partitions of a distributed database. By assigning causally generated timestamps to the transactions based on one or more globally coherent time services, the timestamps can be used to preserve and represent the causal order of the transactions in the distributed system. In this regard, certain transactions may wait for a period of time after choosing a timestamp in order to delay the start of any second transaction that might depend on it. The wait may ensure that the effects of the first transaction are not made visible until its timestamp is guaranteed to be in the past. This may ensure that a consistent snapshot of the distributed database can be determined for any past timestamp.

Ensuring globally consistent transactions

A method of distributing data in a distributed storage system includes receiving a file, dividing the received file into chunks, and determining a distribution of the chunks among storage devices of the distributed storage system based on a maintenance hierarchy of the distributed storage system. The maintenance hierarchy includes maintenance levels, and each maintenance level includes one or more maintenance units. Each maintenance unit has an active state and an inactive state. Moreover, each storage device is associated with a maintenance unit. The determining of the distribution of the chunks includes identifying a random selection of the storage devices matching a number of chunks of the file and being capable of maintaining accessibility of the file when one or more maintenance units are in an inactive state. The method also includes distributing the chunks to storage devices of the distributed storage system according to the determined distribution.

Distributing data on distributed storage systems

The present disclosure relates to dynamically scheduling resource requests in a distributed system based on usage quotas. One example method includes identifying usage information for a distributed system including atoms, each atom representing a distinct item used by users of the distributed system; determining that a usage quota associated with the distributed system has been exceeded based on the usage information, the usage quota representing an upper limit for a particular type of usage of the distributed system; receiving a first request for a particular atom requiring invocation of the particular type of usage represented by the usage quota; determining that a second request for a different type of usage of the particular atom is waiting to be processed; and processing the second request for the particular atom before processing the first request.

Quota-based resource scheduling

Systems and methods for analyzing input data records are provided in which a master process initiates a plurality of concurrent first processes each of which comprises, for each data record in at least a subset of a plurality of input data records, creating a parsed representation of the data record and independently applying a procedural language query to the parsed representation to extract one or more values. A respective emit operator is applied to at least one of the extracted one or more values thereby adding corresponding information to a respective intermediate data structure. The respective emit operator implements one of a predefined set of statistical information processing functions. The master process also initiates a plurality of second processes each of which aggregates information from a corresponding subset of intermediate data structures to produce aggregated data that is, in turn, combined to produce output data.

System and method for analyzing data records

Growing community of inventors

Palo Alto, CA, United States of America

Sean Quinlan