Amazon Technologies, Inc.

Wells Lin

Christopher Magee Greenwood

Kiran-Kumar Muniswamy-Reddy

Danny Wei

Mitchell Gannon Flaherty

Igor A Kostic

Sriram Venugopal

Colin Williams

Yilin Guo

Patrick E Brennan

Linfeng Yu

Divya Ashok Kumar Jain

Gary Michael Herndon

William Zaharchuk

Rahul Upadhyaya

Romain Benoit Seguy

Norbert Paul Kusters

Alexander R Yee

A block storage service can ensure volumes are placed in a same region as an attached virtual machine instance for performance and durability guarantees. A region can reference multiple things, but one example is that a volume is within a same spine as a virtual machine in order to meet performance guarantees. Each region can have a buffer of server computers held in reserve for volumes having a type where performance guarantees are required. If performance guarantees cannot be met, a rejection is transmitted to the customer. In another embodiment, the customer can provide a list in priority order of different volume types so that if a desired volume type cannot be placed, then alternative volume types can be used.

Block storage with volume locality placement based on performance requirements

The present disclosure generally relates to creating virtualized block storage devices whose data is replicated across isolated computing systems to lower risk of data loss even in wide-scale events, such as natural disasters. The virtualized device can include at least two volumes, each of which is implemented in a distinct computing system. Each volume can be encrypted with a distinct key, and an encryption service can operate to transform data 'in-flight' on the replication path between the volumes, reencrypting data according to the key appropriate for each volume.

Encrypted cross-zone replication for cross-zone replicated block storage devices

The present disclosure generally relates to creating virtualized block storage devices whose data is replicated across isolated computing systems to lower risk of data loss even in wide-scale events, such as natural disasters. The virtualized device can include at least two volumes, each of which is implemented in a distinct computing system. Each volume can be implemented by at least two computing devices, a first of which is configured as a primary device to which reads from and writes to the volume are directed. Of the two volumes, one can be indicated as primary, indicating authority to accept reads to and writes from the virtualized device. A primary device of the primary volume, on obtaining a write to the volume, can replicate the write to both a secondary device of a primary volume and to the secondary volume.

Cross-zone replicated block storage devices

The present disclosure generally relates to creating virtualized block storage devices whose data is replicated across isolated computing systems to lower risk of data loss even in wide-scale events, such as natural disasters. The virtualized device can include at least two volumes, each of which is implemented in a distinct computing system. In the case of a failed volume, a new volume can be created and populated with data from the surviving volume. During population, new writes can continue to be replicated to the new volume. The population process can write data from the surviving volume to the new volume 'under' new writes, such that the population process does not overwrite data included in the new writes.

Failover for failed secondary in cross-zone replicated block storage devices

The present disclosure generally relates to creating virtualized block storage devices whose data is replicated across isolated computing systems to lower risk of data loss even in wide-scale events, such as natural disasters. The virtualized device can include at least two volumes, each of which is implemented in a distinct computing system. Each volume can be implemented by at least two computing devices, a first of which is configured as a primary device to which reads from and writes to the volume are directed. To ensure consistency in the distributed device, a multi-tier authority service is implemented, in which a cross-computing system authority service designates a volume as having authority to accept writes to the virtualized device, and in which a second tier authority service designates a computing device as having authority to accept writes to the volume.

Hierarchical authority store for cross-zone replicated block storage devices

Generally described, one or more aspects of the present application correspond to a highly distributed replica of a volume stored in a networked elastic computing environment. First and second replicas of the volume can be synchronously replicated, and some implementations of the tertiary replica can be asynchronously replicated. The highly distributed nature of the tertiary replica supports parallel data transfer of the data of the volume, resulting in faster creation of backups and new copies of the volume.

Rapid volume backup generation from distributed replica

A snapshot object or other type of object may be stored in a first storage system and may be accelerated in another storage system, wherein an accelerated snapshot or other type of object can be used to populate volumes in the other storage system with data more rapidly than a non-accelerated snapshot or other type of object. The accelerated snapshot or other object may be implemented using an intermediate volume implemented in the other storage system that is populated with data from the snapshot object or the other object stored in the first storage system.

Accelerated volumes

Systems and methods are disclosed for handling requests to create multiple volumes with an expected usage correlation on a block storage service. Rather than handling each request to create a volume independently, embodiments described herein can handle the request in bulk, quickly determining whether the request as a whole can be handled at the system. In one embodiment, the service allows for oversubscription of use on hosts, as well as parallelization of placement decisions, by rejecting placement requests when a number of possible hosts for a volume within a requested set falls below a candidate set size, which is determined based on factors such as a number of requested volumes and an expected accuracy of state information for the service at a placement engine within a distributed set of engines.

Correlated volume placement in a distributed block storage service

Systems and methods are disclosed for handling requests to create multiple volumes with an expected usage correlation on a block storage service. Rather than handling each request to create a volume independently, embodiments described herein can handle the request in bulk. In one embodiment, the service allows for oversubscription of use on hosts, as well as parallelization of placement decisions, by distributing requests among a set of parallelized placement engines. Each engine can distribute its subset of volumes at least partly randomly among a candidate set of volumes, with the size of the candidate set selected based on a total number of volumes. This distribution mechanism can ensure distribution of volumes without requiring centralized placement of the volumes.

Generally described, aspects of the present application correspond to enabling rapid duplication of data within a data volume hosted on a network storage system. The network storage system can maintain a highly distributed replica of the data volume, designated for duplication of data within the volume and separate from one or more other replicas designated for handling modifications to the data volume. By providing increased parallelization, the highly distributed replica can facilitate rapid duplication of the volume. When a sufficiently large request to duplicate the data volume is received, the system can create additional duplicate portions of the volume to further increase parallelization. For example, a partition of the highly distributed replica may be repeatedly duplicated to create a large number of intermediary duplicate partitions. The intermediary duplicate partitions can then be used to service the duplication request rapidly, due to increased parallelism.

Intermediary duplication to facilitate copy requests in distributed storage systems

A new metadata field is described that can be used with requests to store a volume in a block storage service. The metadata field is a parameter included in the request and is a logical association identifier that associates the volume to be stored with other already-stored volumes. Using this logical association identifier, the block storage service can ensure that all volumes having the same identifier are sufficiently spread across server computers. The logical association identifier can be a workload identifier indicating that the volumes are all from a same relational database, application, etc. Target customers are defined as customers that request an action of a service within a cloud environment and the service then makes a request to the block storage service. In this case, the service is a customer of the block storage service and it can pass the target customer identifier as the logical association identifier.

Placement of data volumes in a data center

The present disclosure generally relates to creating virtualized block storage devices whose data is replicated across isolated computing systems to lower risk of data loss even in wide-scale events, such as natural disasters. The virtualized device can include at least two volumes, each of which is implemented in a distinct computing system. Due to separation between volumes, replication lag may occur, in which data persisted to a first volume is not immediately persisted to a second volume. Such lag can increase a potential for data loss in the event that the first volume fails. Embodiments of the present disclosure relate to managing data loss risk by determining an expected maximum difference between the data stored at the two volumes, in a manner that does not require decrypting the data written to the volumes or perfect knowledge of the state of the distributed system at a single point.

Approximating replication lag in cross-zone replicated block storage devices

A distributed system allocates capacity in response to client requests. The system monitors a rate of capacity allocation. Based on the monitored rate and on a set of parameters, the system generates a forecast of capacity available for allocation and determines a time when available capacity will fall below a threshold level. The system adjusts the parameters to cause the predicted time to align with a target time, and then causes the system to be reconfigured according to the adjusted parameters.

Dynamic capacity buffers

Generally described, one or more aspects of the present application correspond to a highly distributed replica of a volume stored in a networked computing environment. First and second replicas of the volume can be synchronously replicated, and some implementations of the tertiary replica can be asynchronously replicated. The highly distributed nature of the tertiary replica supports parallel data transfer of the data of the volume, resulting in faster creation of backups and new copies of the volume.

Growing community of inventors

Seattle, WA, United States of America

Wells Lin