Amazon Technologies, Inc.

Adam Frederick Brock

Colin Laird Lazier

James Caleb Kirschner

Bryan James Donlan

Paul David Franklin

Rishabh Animesh

Umar Farooq

A data storage system implements techniques for efficient retrieval of data stored thereon, using time of upload or another monotonically increasing variable as a key or identifier for the data to be stored and/or retrieved. Data is sorted according to, e.g., upload time, and the data is addressed with respect to time of upload and byte offset within the archive.

Monotonic time-based data storage

A data transfer device is used to augment the capabilities of a data storage system. The data transfer device may be capable of persistently storing data for an indeterminate amount of time, and may be configured to store a portion of a bundle of redundancy coded shards that span between the data transfer device and a data storage system configured to store the remainder of the bundle. Data stored on the data transfer device may be read from and written directly to the data transfer device without transfer of data to the data storage system. If the data transfer device is not available, the remaining shards of the bundle may provide a regenerated, original form of the data.

Local storage clustering for redundancy coded data storage system

A cluster of data transfer devices is used to augment the capabilities of a data storage system. For example, the cluster of data transfer devices may be configured to store a portion of a bundle of redundancy coded shards in a similar fashion as a data storage system. As another example, the cluster may be configured to provide other capabilities incident to the devices used, such as computational capabilities. Data stored on the cluster may be read from and written directly to the cluster without transfer of data to the data storage system. In some embodiments, a connecting entity (such as a customer entity) may interchangeably interface with the data storage system and the cluster, and the requested capabilities may be directed to either in a fashion that is transparent to the requestor.

Hybridized storage operation for redundancy coded data storage systems

Cycled clustering for redundancy coded data storage systems

One or more grids of redundancy coded shards, such as those stored or otherwise represented on grid encoded storage systems, are combinable or extensible. For example, a generator matrix of a redundancy code may be configured so as to have a sufficient number of fields to generate a grid. The generator matrix may initially be used to generate smaller grids, which can be combined into the target grid without re-encoding most or all of the data represented thereon. In some cases, vertically derived shards of the input grids may be combined using, e.g., matrix addition, which may then be directly allocated to the target grid, while data shards and horizontally derived shards may be allocated to the target grid with no further transformation.

Techniques for combining grid-encoded data storage systems

Techniques for encoding data storage systems using grid-encoded data storage systems with non-parity linear redundancy encoding schemes are described herein. A grid of shards with derived shards and data shards is generated that is indexed by a first index and a second index and is configured so that each shard is reproducible from other shards with the same first index and is also reproducible from other shards with the same second index. The grid of shards is further configured so that each data row of the grid of shards has at least two derived shards and at least twice as many data shards as derived shards.

Non-parity in grid encoded data storage systems

Techniques described and suggested herein include systems and methods for optimizing random access characteristics for data archives stored on data storage systems using redundancy coding techniques. For example, redundancy coded shards, which may include identity shards that contain unencoded original data of archives, may be configured such that a variable number of the shards can be leveraged to meet random access requirements for retrieval requests associated with the archives stored and/or encoded therein. Implementing systems may monitor random access rates, capabilities, and burdens, so as to adaptively account for changes to some or all of the monitored parameters.

Growing community of inventors

Seattle, WA, United States of America

Adam Frederick Brock