Amazon Technologies, Inc.

Kerry Quintin Lee

Danny Wei

Tate Andrew Certain

Nandakumar Gopalakrishnan

Marc John Brooker

James Michael Thompson

Madhuvanesh Parthasarathy

Jianhua Fan

John Luther Guthrie, Ii

Varun Marupadi

Yan Valerie Leshinsky

Derek Ernest Denny-Brown, Ii

Jiahua He

Norbert Paul Kusters

Matthew James Eddey

Yi Li

Mahmood Miah

Lakshmi N Pallikila

John Luther Guthrie

James Andrew Trenton Lipscomb

Tarun Goyal

Anand S Gupta

Raghav Vijay Pisolkar

George Shipilov

Fnu Amit Anand Amleshwaram

Benjamin Arthur Hawks

Fiorenzo Cattaneo

Jonathon E B Grieman

Ii John Luther Guthrie

Embodiments of system and method for performing object-modifying commands in an unstructured storage service are described. Embodiments may include utilizing one or more nodes to implement an unstructured storage service including a service interface configured to service data storage requests and data retrieval requests from remote clients. The unstructured storage service is configured to isolate access to each of multiple storage domains of the service to different clients. Embodiments may also include receiving from a given remote client, a request to store a data object to a given storage domain of the service. The request further specifying a command to modify source data for the object prior to storing the object. Embodiments may also include using the unstructured storage service to perform the command without exporting the source data from the unstructured storage service to the given remote client prior to completion of the command.

System and method for performing object-modifying commands in an unstructured storage service

A storage controller is implemented for controlling a storage system. The storage controller may be implemented using a distributed computer system and may include components for servicing client data requests based on the characteristics of the distributed computer system, the client, or the data requests. The storage controller is scalable independently of the storage system it controls. All components of the storage controller, as well as the client, may be virtual or hardware-based instances of a distributed computer system.

Environment-sensitive distributed data management

A system and method for maintaining data integrity during data transformation operations. The system and method include obtaining a message from a set of queues, obtaining, from a first data store, a data object indicated by the message, and generating, at a first host, a set of error detection codes corresponding to a transformation of the data object according to a transformation scheme. The system and method further include, transforming, at a second host different from the first host, the data object according to the transformation scheme into the transformation of the data object, verifying the transformation against the set of error detection codes, and storing the transformation in a second data store.

Ensuring data integrity during large-scale data migration

An opportunistic throttling approach can be used for customers of shared resources in a multi-tenant environment. Each customer can have a respective token bucket with a guaranteed fill rate. When a request is received for an amount of work to be performed by a resource, the corresponding number of tokens are obtained from, or charged against, a global token bucket. If the global bucket has enough tokens, and if the customer has not exceeded a maximum work rate or other such metric, the customer can charge less than the full number of tokens against the customer's token bucket, in order to reduce the number of tokens that need to be taken from the customer bucket. Such an approach can enable the customer to do more work and enable the customer's bucket to fill more quickly as fewer tokens are charged against the customer bucket for the same amount of work.

Multi-tenant throttling approaches

A block-based storage system may implement reducing durability state for a data volume. A determination may be made that storage node replicating write requests for a data volume is unavailable. In response, subsequent write requests may be processed according to a reduced durability state for the data volume such that replication for the data volume may be disabled for the storage node. Write requests may then be completed at a fewer number of storage nodes prior to acknowledging the write request as complete. Durability state for the data volume may be increase in various embodiments. A storage node may be identified and replication operations may be performed to synchronize the current data volume at the storage node with a replica of the data volume maintained at the identified storage node.

Reducing data volume durability state for block-based storage

A computing system recovers volumes in a distributed computing environment while reducing downtime of storage servers. In an embodiment, a storage server contacts a control plane after a storage failure has occurred. If the storage server hosts an authoritative copy of an offline volume, the storage server is requested to restore the volume. Non-authoritative volumes are removed from the storage server and the storage server provides read access to the restored volume while resuming storage services.

Data recovery in a distributed computing environment

A caching system and methodology for data in a memory for faster access to commonly-used data by other applications and computer devices on a network. The memory can include a solid-state drive (SSD) array for the cache memory that has read-bias, in addition to a magnetic hard drive array. The system uses a logical set of slots to hold identifiers for specific groups of data that can be placed into cache memory and each identifier has a usage attribute that changes based upon the usage of the specific group of data and causes the identifier to move within the set of slots and potentially into cache memory.

Cache memory data management using relative access frequency

A block-level data storage system receives a request to delete a data storage volume. As a result, the data storage volume is deleted and the areas comprising the volume are released and reaped. The areas may contain non-zero data within a plurality of data storage chunks that comprise the areas. An area cleaner is configured to zero out the areas for allocation to a newly created data storage volume.

Data storage area cleaner

A snapshot depth for a volume is obtained, with the snapshot depth indicating an upper limit to a number of snapshot updates, where an individual snapshot update of the snapshot updates includes data of the volume that has been modified since a time of storage of a previous snapshot update and an amount, based at least in part on the snapshot depth, of other data of the volume. The number of snapshot updates sufficient to construct a full snapshot of a state of the volume at a particular point in time is determined based at least in part on the snapshot depth, and at least the number of snapshot updates is caused to be stored.

Amortized snapshots

A block-based storage system may implement efficient replication for restoring a data volume from a reduced durability state. A storage node that is not replicating write requests for a data volume may determine that replication for the data volume is to be enabled. A peer storage node may be identified that maintains a stale replica of the data volume. One or more replication operations may be performed to update stale data chunks in the stale replica of the data volume with current data chunks without updating data chunks in the stale replica of the data volume that are current. Stale replicas that are no longer needed may be deleted according timeouts or the amount of stale data in the replica.

Efficient data volume replication for block-based storage

Systems and methods are disclosed that facilitate the collection, transmission and processing of information related to the storage processes initiated in conjunction with virtual machine instances instantiated on a hosted virtual network. A virtual machine manager component, such a hypervisor component, receives a storage request from one or more virtual machine instances hosted on a host computing device. Additionally, the virtual machine manager component can further request via the storage protocol that the storage subsystem should collect and return storage process metric information. If the storage subsystem supports the collection of the storage process metric information, the storage subsystem will provide a response to the storage request and at least some of the requested storage process metric information. The virtual machine manager component can then process the response to the storage request as appropriate and forward any storage process metric information to a storage process metrics management component.

Storage process metrics

Distributed data storage controller

Write optimization for block-based storage performing snapshot operations may be implemented. Write requests for a particular data volume may be received for which a snapshot operation is in progress. A determination may be made as to whether a data chunk of the data volume modified as part of the write request has not yet been stored to a remote snapshot data store as part of the snapshot operation. For a data chunk that is to be modified and that has not yet been stored, the data chunk may be stored in a local in-memory volume snapshot buffer. Once the data chunk is stored in the in-memory volume snapshot buffer, the write request may be performed and acknowledged as complete. The data chunk may be sent to the remote snapshot data store asynchronously with regard to the acknowledgment of the write request.

Optimized write performance at block-based storage during volume snapshot operations

A system and method for using a dead letter queue to regulate host computing systems performing transformation or migration operations. The system and method including reading a message from a queue, the message including one or more references to a set of data objects, with the set of data objects including one or more data objects selected for a transformation operation according to a transformation scheme. The system and method further including performing a mitigation action upon determination that the message is one of a number of unsuccessfully processed messages in the queue that are associated with the computer system, when the number of unsuccessfully processed messages exceed a threshold.

Growing community of inventors

Seattle, WA, United States of America

Kerry Quintin Lee