Amazon Technologies, Inc.

Madhuvanesh Parthasarathy

Marc John Brooker

Tate Andrew Certain

Tobias L Holgers

David R Richardson

Danny Wei

Kerry Quintin Lee

Marc Levy

Yu Li

Tao Chen

Marc Stephen Olson

Christopher Magee Greenwood

Surya Prakash Dhoolam

Divya Ashok Kumar Jain

Fan Ping

Shobha Agrawal

Lakshmi N Pallikila

John Luther Guthrie

Matthew Joseph Zulak

Tarun Goyal

Brian Todd Burruss

Brandon William Porter

Gary Michael Herndon

Joshua Dawie Mentz

Steven J Kruy

Michael Groenewald

Mustafa I Abrar

Mark Danoher

Gabriël De Vos Smit

A storage management system monitors an indicator of whether data storage is or will exceed a threshold of storage utilization as stored on a current implementation resource, such as a storage server. The indicator may be used to determine whether none, some or all of the data storage should be moved from the current implementation resource to an available implementation resource.

Data volume placement techniques

A storage controller is implemented for controlling a storage system. The storage controller may be implemented using a distributed computer system and may include components for servicing client data requests based on the characteristics of the distributed computer system, the client, or the data requests. The storage controller is scalable independently of the storage system it controls. All components of the storage controller, as well as the client, may be virtual or hardware-based instances of a distributed computer system.

Environment-sensitive distributed data management

The relative health of data storage drives may be determined based, at least in some aspects, on data access information and/or other drive operation information. In some examples, upon receiving the operation information from a computing device, a health level of a drive may be determined. The health level determination may be based at least in part on operating information received from a client entity. Additionally, a storage space allocation instruction or operation may be determined for execution. The allocation instruction or operation determined to be performed may be based at least in part on the determined health level.

Server-side, variable drive health determination

Data replication groups may be used to store data in a distributed computing environment. The data replication groups may include a set of nodes executing a consensus protocol to maintain data durably. In order to increase the speed of provisioning, a data replication may be provision with less than the total number of nodes to be included in the group. Once an initial set of nodes have been provisioned and are implementing a consensus protocol, the data replication group may begin to receive traffic. Furthermore, new nodes may be added to the group until the total number of nodes for the group has been reached.

Dynamic provisioning for data replication groups

A customer of a computing resource service provider submits a request to a virtual computer system service to instantiate a virtual computer system. In response to the request, the service obtains physical network information in order to identify any hardware groupings that may be used, at a later time, to provision one or more data storage volumes that may be associated with the virtual computer system. The service assigns a score to each of these hardware groupings based at least in part on the available capacity for provisioning of these data storage volumes. Based at least in part on these assigned scores, the virtual computer system service selects a hardware grouping and instantiates the virtual computer system within the selected hardware grouping.

Hardware grouping based computing resource allocation

Systems and methods are described for a storage processing service that processes multiple storage commands. The storage processing service uses tags from test storage commands to determine whether a test storage service is to be instantiated that reflects a corresponding production service. Test storage commands with the same tag are tested on that test service. Additionally, the storage processing service determines a strategy for testing processes on production services when the storage system is overloaded. In one embodiment, the test service manager can determine to stop testing processes for a period of time, and issue a shed command that queues or sheds test storage commands. Advantageously, a shed command, while active at a storage processing service, may alleviate the overload on production services. The test service manager can continue to monitor the storage system to determine whether the overload continues to exist.

Managing test services in a distributed production service environment

Data replication groups may be used to store data in a distributed computing environment. The data replication groups may include a set of nodes executing a consensus protocol to maintain data durably. In order to increase the speed of provisioning, a pool of data replication groups may be maintained. When a request is received a data replication group may be selected from the pool, removed from the pool, and associated with the request. Furthermore, a replacement data replication group may be provisioned and added to the pool.

Replication group pools for fast provisioning

Variable drive health determination and data placement

A distributed system may implement logical control groups for distributed system resources. A request for a particular system resource may be received at a request routing tier for a distributed system that is implemented by multiple computing systems in different respective availability zones. One or more resource control nodes of a resource control tier for the distributed system that are members of a logical control group for the particular system resource may be discovered. A logical control group for the particular system resource may include at least one resource control node in a different availability zone than the availability zone that includes the particular system resource. The request may then be sent to one of the discovered resource control nodes in the logical control group in order to service the request for the particular system resource.

Logical control groups for distributed system resources

A distributed system may implement dynamic resource creation for connecting resources. Resource creation requests may be received for resources which are accessed by other resources hosted in a distributed system. An intention for the creation request is stored to perform the creation request. At a subsequent time, a request to connect another resource to the resource of the creation request is received. An infrastructure locality for the other resource may then be identified and utilized to determine a placement location for the resource. The resource may then be created at the placement location according to the stored intention. The resource may then be connected with the other resource for access.

Dynamic resource creation to connect client resources in a distributed system

A system may implement maintaining control plane data versions for a network-based service control plane. Various control plane actions may be performed which create new versions of control plane data that may be maintained for the control plane in a database. Some of these actions may be performed by multiple actors creating new versions of the same control plane data. For a particular control plane action, a new version number may be obtained to include in a new version of control plane data, and a conditional write request may be performed to insert the new version of control plane data at the database as part of an optimistic concurrency technique in order to maintain consistency for control plane data.

Maintaining versions of control plane data for a network-based service control plane

Distributed data storage controller

A storage management system monitors relationships between data sets stored on current implementation resources, such as a storage servers. The relationships may be used to determine whether a data set should be moved from a current implementation resource to an available implementation resource.

A storage management system monitors an indicator of whether data storage is or will be sufficiently responsive as stored on a current implementation resource, such as a storage server. The indicator may be used to determine whether none, some or all of the data storage should be moved from the current implementation resource to an available implementation resource.

Growing community of inventors

Renton, WA, United States of America

Madhuvanesh Parthasarathy