Amazon Technologies, Inc.

Divya Ashok Kumar Jain

Fan Ping

Tao Chen

Norbert Paul Kusters

Marc John Brooker

Danny Wei

Kiran-Kumar Muniswamy-Reddy

Lakshmi N Pallikila

William Zaharchuk

Wells Lin

Shobha Agrawal

Rahul Upadhyaya

Thomas Tarak Mathew Veppumthara

Romain Benoit Seguy

James T Pinkerton

Jianhua Fan

Matthew Joseph Zulak

Andrew Boyer

Oleksandr Chychykalo

Sebastiano Peluso

Christopher Magee Greenwood

Madhuvanesh Parthasarathy

Matthew Shawn Wilson

Surya Prakash Dhoolam

Sriram Venugopal

Amol Madhav Joshi

Anish Borkar

John Michael Hayden

Systems and methods are provided for hosting block storage volumes in a hosted computing environment that are supported by multiple underlying elements and that are distributed among physical host devices in a manner that provides for resiliency of the volumes in instances of power failure. As disclosed herein, a block storage volume may be supported by a combination of underlying elements, such as replicas that store data of the volume and replicated authority nodes that designate authority of a replica to accept writes to the volume. To ensure resiliency to power failure, both replicas and authority nodes can be distributed among physical host devices that are supplied by independent power supplies, such that failure of one independent power supply does not render the volume non-operational.

Power-diverse placement for data replicas and authority nodes in network-based storage

A block-based storage system hosts logical volumes that are implemented via multiple replicas of volume data stored on multiple resource hosts in different failure domains. Also, the block-based storage service allows multiple client computing devices to attach to a same given logical volume at the same time. In order to prevent unnecessary failovers, a primary node storing a primary replica is configured with a health check application programmatic interface (API) and a secondary node storing a secondary replica determines whether or not to initiate a failover based on the health of the primary replica.

Block-storage service supporting multi-attach and health check failover mechanism

Methods, systems, and computer-readable media for distributed transactions across multiple consensus groups are disclosed. A distributed transaction system comprises a proposer and a plurality of consensus groups, including a first consensus group comprising a first plurality of members and a second consensus group comprising a second plurality of members. The proposer proposes a transaction to at least a portion of the first consensus group and at least a portion of the second consensus group. A majority of the members in the first consensus group agree to perform the transaction, where the transaction is selected from a plurality of proposed transactions involving the first consensus group. A majority of the members in the second consensus group also agree to perform the transaction. The members of the first and second consensus groups perform the transaction to update a plurality of stored replicas.

Distributed transactions across multiple consensus groups

The present disclosure generally relates to creating virtualized block storage devices whose data is replicated across isolated computing systems to lower risk of data loss even in wide-scale events, such as natural disasters. The virtualized device can include at least two volumes, each of which is implemented in a distinct computing system. Each volume can be encrypted with a distinct key, and an encryption service can operate to transform data 'in-flight' on the replication path between the volumes, reencrypting data according to the key appropriate for each volume.

Encrypted cross-zone replication for cross-zone replicated block storage devices

The present disclosure generally relates to creating virtualized block storage devices whose data is replicated across isolated computing systems to lower risk of data loss even in wide-scale events, such as natural disasters. The virtualized device can include at least two volumes, each of which is implemented in a distinct computing system. Each volume can be implemented by at least two computing devices, a first of which is configured as a primary device to which reads from and writes to the volume are directed. Of the two volumes, one can be indicated as primary, indicating authority to accept reads to and writes from the virtualized device. A primary device of the primary volume, on obtaining a write to the volume, can replicate the write to both a secondary device of a primary volume and to the secondary volume.

Cross-zone replicated block storage devices

Data replication groups may be used to store data in a distributed computing environment. A data replication group may include a set of nodes executing a consensus protocol to maintain data durably. In order to increase efficiency and performance of the data replication group leadership and/or fitness information may be determined for each node of the set of nodes executing the consensus protocol. The fitness information may be based at least in part on information associated with computing resource in the distributed computing environment and one or more constraints on assignment of a role of master node in the data replication group. The master node may be responsible for advancing the data replication group.

Prioritized leadership for data replication groups

Updates for clients accessing a data volume across fault tolerance zones may be replicated. Requests to write or read a block of a logical volume with replicas located in different fault tolerance zones may be sent to the different replicas. Responses for the different requests may be evaluated to determine whether quorum is satisfied for the write or read of the block of the logical volume. For writes that satisfy quorum, a request to commit the write may be sent to the replicas of the logical volume.

Replicating data volume updates from clients accessing the data volume across fault tolerance zones

The present disclosure generally relates to creating virtualized block storage devices whose data is replicated across isolated computing systems to lower risk of data loss even in wide-scale events, such as natural disasters. The virtualized device can include at least two volumes, each of which is implemented in a distinct computing system. In the case of a failed volume, a new volume can be created and populated with data from the surviving volume. During population, new writes can continue to be replicated to the new volume. The population process can write data from the surviving volume to the new volume 'under' new writes, such that the population process does not overwrite data included in the new writes.

Failover for failed secondary in cross-zone replicated block storage devices

The present disclosure generally relates to creating virtualized block storage devices whose data is replicated across isolated computing systems to lower risk of data loss even in wide-scale events, such as natural disasters. The virtualized device can include at least two volumes, each of which is implemented in a distinct computing system. Each volume can be implemented by at least two computing devices, a first of which is configured as a primary device to which reads from and writes to the volume are directed. To ensure consistency in the distributed device, a multi-tier authority service is implemented, in which a cross-computing system authority service designates a volume as having authority to accept writes to the virtualized device, and in which a second tier authority service designates a computing device as having authority to accept writes to the volume.

Hierarchical authority store for cross-zone replicated block storage devices

The present disclosure generally relates to creating virtualized block storage devices whose data is replicated across isolated computing systems to lower risk of data loss even in wide-scale events, such as natural disasters. The virtualized device can include at least two volumes, each of which is implemented in a distinct computing system. Due to separation between volumes, replication lag may occur, in which data persisted to a first volume is not immediately persisted to a second volume. Such lag can increase a potential for data loss in the event that the first volume fails. Embodiments of the present disclosure relate to managing data loss risk by determining an expected maximum difference between the data stored at the two volumes, in a manner that does not require decrypting the data written to the volumes or perfect knowledge of the state of the distributed system at a single point.

Approximating replication lag in cross-zone replicated block storage devices

Generally described, one or more aspects of the present application correspond to techniques for dynamically determining token bucket sizes for different types of transactions occurring at a node in a distributed consensus-based transaction system. For example, each node can store a number of cells, with such cells being replicated across a number of nodes. Each cell can assume various roles in consensus transactions, for example being a leader or simple juror, or being unsure of its leader's identity and requesting leadership. Each node can determine the overall number of transaction tokens that it will assign during a given time period based on the cell composition, and can allocate those tokens to a leader election token bucket and a customer traffic token bucket based on the cell composition.

Dynamic throttling for data consensus groups

A block-based storage system hosts logical volumes that are implemented via multiple replicas of volume data stored on multiple resource hosts in different failure domains. Also, the block-based storage service allows multiple client computing devices to attach to a same given logical volume at the same time. A membership group authority authorizes sequence numbers for a given logical volume and an associated membership group. The members of the membership group ensure that the members are in agreement on the latest sequence number for the given logical volume before responding to read or write requests directed to the given logical volume.

Block-storage service supporting multi-attach

Data replication groups may be used to store data in a distributed computing environment. The data replication groups may include a set of nodes executing a consensus protocol to maintain data durably. The nodes may be executed by one or more host computer systems. In order to avoid failure of the data replication groups as safe deployment strategy is provided. The one or more host computer systems may be assigned to deployment groups where the number of hosts assigned to the deployment groups host less than a quorum of the nodes of a particular data replication group.

Growing community of inventors

Issaquah, WA, United States of America

Divya Ashok Kumar Jain