Amazon Technologies, Inc.

Shobha Agrawal

Marc John Brooker

Tao Chen

Lakshmi N Pallikila

Divya Ashok Kumar Jain

Fan Ping

Matthew Joseph Zulak

David R Richardson

Danny Wei

Norbert Paul Kusters

John Luther Guthrie, Ii

Nachiappan Arumugam

Christopher Nathan Watson

Varun Verma

Rucha Nene

Daniel Rabinovich

Josephine Reiko Gibney

Andre Podnozov

Shreyas Ramalingam

Ravi Nankani

Sandeep Kumar

Madhuvanesh Parthasarathy

Suresh Babu Sajja

Sahil Doshi

Shuvabrata Ganguly

A data storage system includes a head node and mass storage devices. The head node is configured to flush data stored in a storage of the head node to a set of the mass storage devices of the data storage system. A head node may flush both current version data and point-in-time version data to the set of mass storage devices. Also, the data storage system maintains an index that indicates storage locations of data for particular portions of a volume before and after the data is flushed to the set of mass storage devices. In some embodiments an index includes a current version reference for a volume or portion of a volume and one or more point-in-time snapshot references.

Data system with flush views

Policies can be applied to, and enforced for, specific resources by applying a corresponding tag to those resources. An entity, such as a customer of a resource provider, can generate one or more policies to be applied to a set of resources, where those policies can relate to data retention, backup, lifecycle events, and other such aspects. Each policy can be associated with a particular tag, which may comprise a key-value pair to be applied to various resources. A policy enforcement manager can determine the tagged resources and ensure that the relevant policies are applied. The policies can include logic or intelligence for performing a variety of tasks with respect to resources, groups of resources, or types of resources, as identified using the tags.

Resource lifecycle automation

Data replication groups may be used to store data in a distributed computing environment. The data replication groups may include a set of nodes executing a consensus protocol to maintain data durably. The nodes of the data replication groups may generate logs containing information corresponding committed operations performed by the nodes. These logs may be collected and processed to obtain useful information corresponding to the operation of the data replication group. Furthermore, this processed information may be provided in the form of a stream to enable event driven operations corresponding to the logs.

Distributed log processing for data replication groups

Generally described, one or more aspects of the present application relate to a public snapshot service for creating and managing block-level snapshots on a cloud provider network. Storage locations for each block that comprise the snapshot can be enumerated on a snapshot manifest. Identification of storage location for a requested snapshot block can be improved using a manifest index.

Manifest index for block-level snapshots

Generally described, one or more aspects of the present application relate to a public snapshot service for creating and managing block-level snapshots. Snapshots can be configured to enable writing at the sub-block level. Subsequent modifications made to blocks storage volume that forms the basis of the snapshots can be backed up as a sub-block level operations.

Sub-block modifications for block-level snapshots

Data replication groups may be used to store data in a distributed computing environment. The data replication groups may include a set of nodes executing a consensus protocol to maintain data durably. The nodes may be executed by one or more host computer systems. In order to avoid failure of the data replication groups as safe deployment strategy is provided. The one or more host computer systems may be assigned to deployment groups where the number of hosts assigned to the deployment groups host less than a quorum of the nodes of a particular data replication group.

Deployment strategy for maintaining integrity of replication groups

Data replication groups may be used to store data in a distributed computing environment. The data replication groups may include a set of nodes executing a consensus protocol to maintain data durably. In order to increase the speed of provisioning, a data replication may be provision with less than the total number of nodes to be included in the group. Once an initial set of nodes have been provisioned and are implementing a consensus protocol, the data replication group may begin to receive traffic. Furthermore, new nodes may be added to the group until the total number of nodes for the group has been reached.

Dynamic provisioning for data replication groups

Data replication groups may be used to store data in a distributed computing environment. The data replication groups may include a set of nodes executing a consensus protocol to maintain data durably. During the execution of the set of nodes various nodes may become stale or otherwise obtain a state that is inconsistent with at least one other node of the data replication group. A catch up algorithm may be employed in which a set of teachers is initialized, the various node which may be stale may select a teacher from the set of teachers and perform learning operations. This process may be repeated until the state of the various nodes is current with at least one other node of the data replication group.

Unsupervised round robin catch up algorithm

A data storage system includes a head node and mass storage devices. The head node is configured to flush data stored in a storage of the head node, based at least in part on one or more triggers being met, from the storage of the head node to a set of the mass storage devices of the data storage system. The flushed data is written to a segment of free storage space across the set of the mass storage devices allocated for the given data flush operation. In some embodiments, a head node may flush both current version data and point-in-time version data to the set of mass storage devices. Also, the data storage system maintains an index that indicates storage locations of data for particular portions of a volume before and after the data is flushed to the set of mass storage devices.

Data system with data flush mechanism

Data replication groups may be used to store data in a distributed computing environment. The data replication groups may include a set of nodes executing a consensus protocol to maintain data durably. In order to monitor and debug the operation of the data replication group a chaining mechanism may be utilized during log creation by the set of nodes of the data replication group. The chaining mechanism may cause entries in the log to indicate an operation being performed and an operation performed immediately prior to the operation being performed. In various embodiments, an outside observer receives the logs and checks the logs for errors indicated by the chaining mechanism.

Chaining log operations in data replication groups

Data replication groups may be used to store data in a distributed computing environment. The data replication groups may include a set of nodes executing a consensus protocol to maintain data durably. In order to increase the speed of provisioning, a pool of data replication groups may be maintained. When a request is received a data replication group may be selected from the pool, removed from the pool, and associated with the request. Furthermore, a replacement data replication group may be provisioned and added to the pool.

Growing community of inventors

Sammamish, WA, United States of America

Shobha Agrawal