Amazon Technologies, Inc.

Fan Ping

Divya Ashok Kumar Jain

Tao Chen

Marc John Brooker

Lakshmi N Pallikila

Norbert Paul Kusters

James T Pinkerton

Jianhua Fan

Shobha Agrawal

Matthew Joseph Zulak

Danny Wei

Andrew Boyer

Thomas Tarak Mathew Veppumthara

Oleksandr Chychykalo

Sebastiano Peluso

Madhuvanesh Parthasarathy

Sriram Venugopal

Matthew Shawn Wilson

Avram Israel Blaszka

Anish Borkar

Divya Ashokkumar Jain

John Michael Hayden

Systems and methods are provided for hosting block storage volumes in a hosted computing environment that are supported by multiple underlying elements and that are distributed among physical host devices in a manner that provides for resiliency of the volumes in instances of power failure. As disclosed herein, a block storage volume may be supported by a combination of underlying elements, such as replicas that store data of the volume and replicated authority nodes that designate authority of a replica to accept writes to the volume. To ensure resiliency to power failure, both replicas and authority nodes can be distributed among physical host devices that are supplied by independent power supplies, such that failure of one independent power supply does not render the volume non-operational.

Power-diverse placement for data replicas and authority nodes in network-based storage

A block-based storage system hosts logical volumes that are implemented via multiple replicas of volume data stored on multiple resource hosts in different failure domains. Also, the block-based storage service allows multiple client computing devices to attach to a same given logical volume at the same time. In order to prevent unnecessary failovers, a primary node storing a primary replica is configured with a health check application programmatic interface (API) and a secondary node storing a secondary replica determines whether or not to initiate a failover based on the health of the primary replica.

Block-storage service supporting multi-attach and health check failover mechanism

Methods, systems, and computer-readable media for distributed transactions across multiple consensus groups are disclosed. A distributed transaction system comprises a proposer and a plurality of consensus groups, including a first consensus group comprising a first plurality of members and a second consensus group comprising a second plurality of members. The proposer proposes a transaction to at least a portion of the first consensus group and at least a portion of the second consensus group. A majority of the members in the first consensus group agree to perform the transaction, where the transaction is selected from a plurality of proposed transactions involving the first consensus group. A majority of the members in the second consensus group also agree to perform the transaction. The members of the first and second consensus groups perform the transaction to update a plurality of stored replicas.

Distributed transactions across multiple consensus groups

Data replication groups may be used to store data in a distributed computing environment. The data replication groups may include a set of nodes executing a consensus protocol to maintain data durably. The nodes of the data replication groups may generate logs containing information corresponding committed operations performed by the nodes. These logs may be collected and processed to obtain useful information corresponding to the operation of the data replication group. Furthermore, this processed information may be provided in the form of a stream to enable event driven operations corresponding to the logs.

Distributed log processing for data replication groups

Systems and methods are provided to manage replicas of a virtualized block storage volume. The master replica of the virtualized block storage volume can heartbeat with each secondary replica of the virtualized block storage volume to provide an indication of the status of the master replica. Each secondary replica can reply to the heartbeat of the master replica. Each replica can be configured to request an updated replica configuration based on not receiving a heartbeat from one of the replicas. The master replica can request an updated replica configuration after a first time period without receiving a reply from one of the secondary replicas and each secondary replica can request an updated replica configuration after a second time period without receiving a communication from the master replica. Use of the heartbeat process between the master replica and the secondary replicas can increase system speed or reduce power consumption.

Consensus-based authority selection in replicated network-accessible block storage devices

A system that hosts computing resources may implement optimistically granting permission to host computing resources. A request for permission to host a computing resource may be received by a control plane. If the control plane determines that the resource host is the first to request permission to host the resource, then the control plane may store an indication of permission that blocks other resource hosts from obtaining permission to host the computing resource and sending an acknowledgement of permission to the resource host that requested permission.

Optimistically granting permission to host computing resources

A block-based storage system hosts logical volumes that are implemented via multiple replicas of volume data stored on multiple resource hosts in different failure domains. Also, the block-based storage service allows multiple client computing devices to attach to a same given logical volume at the same time. A membership group authority authorizes sequence numbers for a given logical volume and an associated membership group. The members of the membership group ensure that the members are in agreement on the latest sequence number for the given logical volume before responding to read or write requests directed to the given logical volume.

Block-storage service supporting multi-attach

Data replication groups may be used to store data in a distributed computing environment. The data replication groups may include a set of nodes executing a consensus protocol to maintain data durably. In order to increase the speed of provisioning, a data replication may be provision with less than the total number of nodes to be included in the group. Once an initial set of nodes have been provisioned and are implementing a consensus protocol, the data replication group may begin to receive traffic. Furthermore, new nodes may be added to the group until the total number of nodes for the group has been reached.

Dynamic provisioning for data replication groups

Data replication groups may be used to store data in a distributed computing environment. A data replication group may include a set of nodes executing a consensus protocol to maintain data durably. In order to increase efficiency and performance of the data replication, a particular node of the data replication group may be assigned the role of master node. The role of master node may be lease in accordance with a consensus protocol. If the lease is not renewed within an interval of time election/selection of a new master node may be commenced.

Leadership lease protocol for data replication groups

Data replication groups may be used to store data in a distributed computing environment. The data replication groups may include a set of nodes executing a consensus protocol to maintain data durably. During the execution of the set of nodes various nodes may become stale or otherwise obtain a state that is inconsistent with at least one other node of the data replication group. A catch up algorithm may be employed in which a set of teachers is initialized, the various node which may be stale may select a teacher from the set of teachers and perform learning operations. This process may be repeated until the state of the various nodes is current with at least one other node of the data replication group.

Unsupervised round robin catch up algorithm

Data replication groups may be used to store data in a distributed computing environment. The data replication groups may include a set of nodes executing a consensus protocol to maintain data durably. In order to monitor and debug the operation of the data replication group a chaining mechanism may be utilized during log creation by the set of nodes of the data replication group. The chaining mechanism may cause entries in the log to indicate an operation being performed and an operation performed immediately prior to the operation being performed. In various embodiments, an outside observer receives the logs and checks the logs for errors indicated by the chaining mechanism.

Chaining log operations in data replication groups

Data replication groups may be used to store data in a distributed computing environment. The data replication groups may include a set of nodes executing a consensus protocol to maintain data durably. In order to increase the speed of provisioning, a pool of data replication groups may be maintained. When a request is received a data replication group may be selected from the pool, removed from the pool, and associated with the request. Furthermore, a replacement data replication group may be provisioned and added to the pool.

Growing community of inventors

Kenmore, WA, United States of America

Fan Ping