Google Inc.

Wilson C Hsieh

Sanjay Ghemawat

Jeffrey Adgate Dean

Michael Burrows

Andrew Fikes

Deborah Anne Wallach

Fay Wen Chang

Andrew B Fikes

Updates to a plurality of data sets may be recorded in a shared update log. Whenever a server responsible for the plurality of data sets fails, other servers may reconstruct the data sets using the shared update log and redundant copies of the data sets. As part of the reconstruction process, the other servers reorganize (e.g., by sorting) the update records of sub-logs of the update log to group update records corresponding to the same data set together. The updates are applied to the redundant copies of the data set snapshots to reconstruct the data set.

Data reconstruction from shared update log

Each of a plurality of data items is stored in a table data structure. The table structure includes a plurality of columns. Each of the columns is associated with one of a plurality of locality groups. Each locality group is stored as one or more corresponding locality group files that include the data items in the columns associated with the respective locality group. In some embodiments, the columns of the table data structure may be grouped into groups of columns and each group of columns is associated with one of a plurality of locality groups. Each locality group is stored as one or more corresponding locality group files that include the data items in the group of columns associated with the respective locality group.

Storing a sparse table using locality groups

Each of a plurality of data items is stored in a table data structure. A row identifier and column identifier are associated with each respective data item, and each respective item is stored at a logical location in the table data structure specified by its row identifier and column identifier. A plurality of data items is stored in a cell of the table data structure, and a timestamp is associated with each of the plurality of data items stored in the cell. Each of the data items stored in the cell has the same row identifier, the same column identifier, and a distinct timestamp. In some embodiments, each row identifier is a string of arbitrary length and arbitrary value. Similarly, in some embodiments each column identifier is a string of arbitrary length and arbitrary value.

Large scale data storage in sparse tables

Growing community of inventors

New York, NY, United States of America

Wilson C Hsieh