Amazon Technologies, Inc.

Sriram Venugopal

Christopher Magee Greenwood

Mitchell Gannon Flaherty

Gary Michael Herndon

Wells Lin

Linfeng Yu

Patrick E Brennan

Yilin Guo

Danny Wei

Sandeep Kumar

Lalit Jain

Arvind Chandrasekar

James T Pinkerton

Surya Prakash Dhoolam

Jianhua Fan

Varun Verma

Fan Ping

Kristina Kraemer Brenneman

Oscar Allen Grim Courchaine

Alexander R Yee

Rahul Karnik

Norbert Paul Kusters

Matthew Shawn Wilson

Marc Stephen Olson

Divya Ashok Kumar Jain

Kun Tang

Avram Israel Blaszka

Mark Robinson

Divya Ashokkumar Jain

Vivek Ramchandra Kumkar

Anish Borkar

John Michael Hayden

Systems and methods are provided for hosting block storage volumes in a hosted computing environment that are supported by multiple underlying elements and that are distributed among physical host devices in a manner that provides for resiliency of the volumes in instances of power failure. As disclosed herein, a block storage volume may be supported by a combination of underlying elements, such as replicas that store data of the volume and replicated authority nodes that designate authority of a replica to accept writes to the volume. To ensure resiliency to power failure, both replicas and authority nodes can be distributed among physical host devices that are supplied by independent power supplies, such that failure of one independent power supply does not render the volume non-operational.

Power-diverse placement for data replicas and authority nodes in network-based storage

A block storage service analyzes customer behaviors as external signals to the health of the block storage service. In one example, using rules set by a customer for a health monitoring service, the block storage service can gain insights into what the customer cares about. The customer rules can then be used to setup internal monitors within the block storage service. If the internal monitors are triggered, the block storage service can proactively correct problems before the customer alarm thresholds are reached. For example, customer volumes can be moved to increase performance characteristics. In some cases, if the customer has too much performance capability, the customer volume can be moved to a less costly alternative.

Using customer input to determine proper functionality of a block storage service

A block storage service can ensure volumes are placed in a same region as an attached virtual machine instance for performance and durability guarantees. A region can reference multiple things, but one example is that a volume is within a same spine as a virtual machine in order to meet performance guarantees. Each region can have a buffer of server computers held in reserve for volumes having a type where performance guarantees are required. If performance guarantees cannot be met, a rejection is transmitted to the customer. In another embodiment, the customer can provide a list in priority order of different volume types so that if a desired volume type cannot be placed, then alternative volume types can be used.

Block storage with volume locality placement based on performance requirements

A data storage system includes a head node and mass storage devices. The head node is configured to store volume data and flush volume data to the mass storage devices. Additionally, the head node is configured to determine a quantity of data partitions and/or parity partitions to store for a chunk of volume data being flushed to the mass storage devices in order to satisfy a durability guarantee. For chunks of data for which complete copies are also stored in an additional data storage system, the head node is configured to reduce the quantity of data partitions and/or parity partitions stored such that required storage space is reduced while still ensuring that the durability guarantee is satisfied.

Durable data storage with snapshot storage space optimization

Generally described, one or more aspects of the present application correspond to techniques for creating encrypted block store volumes of data from unencrypted object storage snapshots of the volumes. These encryption techniques use a special pool of servers for performing the encryption. These encryption servers are not accessible to users, and they perform encryption and pass encrypted volumes to other block store servers for user access. The encryption context for the volumes can be persisted on the encryption severs for as long as needed for encryption and not shared with the user-facing servers in order to prevent user access to encryption context.

Encryption by default in an elastic computing system

Systems and methods are described for implementing load-dependent encryption mechanism selection in an elastic computing system. The elastic computing system can include a set of host devices configured to implement block storage volumes on behalf of users. Users may desire that such volumes be encrypted prior to storing data. It may be generally preferable for encryption to occur on the same host devices that host the volume, to reduce latency and bandwidth usage needed to encrypt the data. However, encryption of data can utilize significant computational resources, which may not be available on host devices that also have sufficient storage resources to host the volume. The present disclosure describes systems and methods that can account for computational resource availability on host devices, selecting 'in-place' encryption only when available resources exist on host devices, and otherwise implementing remote encryption of volume data.

Load-dependent encryption mechanism selection in an elastic computing system

A data storage system includes multiple data storage devices. A subset of the data storage devices are selected to implement log storages for the data storage system, wherein incoming read and write requests are serviced at the data storage devices implementing the log storages. Data written to a volume stored in the data storage system is initially written to the log storage and subsequently flushed to additional data storage implemented using remaining ones of the data storage devices of the data storage system. A controller monitors wear levels of the data storage devices and initiates a reorganization of which data storage devices implement the log storages and which data storage devices implement the additional storage such that discrepancies in wear between the data storage devices is reduced.

Data storage system with uneven drive wear reduction

A system that hosts computing resources may implement optimistically granting permission to host computing resources. A request for permission to host a computing resource may be received by a control plane. If the control plane determines that the resource host is the first to request permission to host the resource, then the control plane may store an indication of permission that blocks other resource hosts from obtaining permission to host the computing resource and sending an acknowledgement of permission to the resource host that requested permission.

Optimistically granting permission to host computing resources

Systems and methods are disclosed for handling requests to create multiple volumes with an expected usage correlation on a block storage service. Rather than handling each request to create a volume independently, embodiments described herein can handle the request in bulk, quickly determining whether the request as a whole can be handled at the system. In one embodiment, the service allows for oversubscription of use on hosts, as well as parallelization of placement decisions, by rejecting placement requests when a number of possible hosts for a volume within a requested set falls below a candidate set size, which is determined based on factors such as a number of requested volumes and an expected accuracy of state information for the service at a placement engine within a distributed set of engines.

Correlated volume placement in a distributed block storage service

Systems and methods are disclosed for handling requests to create multiple volumes with an expected usage correlation on a block storage service. Rather than handling each request to create a volume independently, embodiments described herein can handle the request in bulk. In one embodiment, the service allows for oversubscription of use on hosts, as well as parallelization of placement decisions, by distributing requests among a set of parallelized placement engines. Each engine can distribute its subset of volumes at least partly randomly among a candidate set of volumes, with the size of the candidate set selected based on a total number of volumes. This distribution mechanism can ensure distribution of volumes without requiring centralized placement of the volumes.

A new metadata field is described that can be used with requests to store a volume in a block storage service. The metadata field is a parameter included in the request and is a logical association identifier that associates the volume to be stored with other already-stored volumes. Using this logical association identifier, the block storage service can ensure that all volumes having the same identifier are sufficiently spread across server computers. The logical association identifier can be a workload identifier indicating that the volumes are all from a same relational database, application, etc. Target customers are defined as customers that request an action of a service within a cloud environment and the service then makes a request to the block storage service. In this case, the service is a customer of the block storage service and it can pass the target customer identifier as the logical association identifier.

Placement of data volumes in a data center

Generally described, one or more aspects of the present application correspond to techniques for creating multiple encrypted block store volumes of data from an unencrypted source. These encryption techniques can use a transform fleet as an intermediary use between the unencrypted source and the encrypted volumes. The transform fleet can obtain data of the volume from one or both of two sources&#x2014;an object storage 'snapshot' a block storage 'source volume'&#x2014;and can then apply the appropriate encryption key for performing the encryption of a particular volume.

Creating multiple encrypted volumes from a single source

A distributed system allocates capacity in response to client requests. The system monitors a rate of capacity allocation. Based on the monitored rate and on a set of parameters, the system generates a forecast of capacity available for allocation and determines a time when available capacity will fall below a threshold level. The system adjusts the parameters to cause the predicted time to align with a target time, and then causes the system to be reconfigured according to the adjusted parameters.

Dynamic capacity buffers

Data volumes hosted for customers in a multi-tenant environment can be moved advantageously throughout the environment to improve performance and reduce cost. A data volume can serve I/O for a virtual machine instance, and it can be advantageous for the virtual machine and the data volume to be in the same network locality, or share at least some of the same network state and interconnection. Since there is limited capacity in a network locality, data volumes not attached to virtual machines can be moved to other locations in the environment. This can include moving data volumes to other local network topologies or snapshotting data volumes and writing the snapshots to another storage service. If the data volume is again needed for I/O, the data volume can moved, allocated, or reattached as necessary.

Growing community of inventors

Issaquah, WA, United States of America

Sriram Venugopal