Amazon Technologies, Inc.

Microsoft Technology Licensing, LLC

Mark Christopher Seigle

Colin Laird Lazier

Kestutis Patiejunas

Bryan James Donlan

Christian L Claiborn

Kamran Tirdad

Gregory Branchek Roth

Gregory Alan Rubin

Sandeep Kumar

Claire Elizabeth Suver

Ryan Charles Schmitt

Jeffrey Arthur Johnson

James Christopher Sorenson

Claire E Suver Weir

James Robert Hamilton

Manuvir Das

Sandesh Doddameti

Rishabh Animesh

Alyssa H Henry

Evelina Colleen Babcock

Weixiang Sun

Sorin L Lerner

Claire E Suver

James Christopher Sorenson Iii

An application programming interface for a data storage service provides a convenient mechanism for clients of the data storage service to access its various capabilities. An API call may be made to initiate a job and in response a job identifier may be provided. A separate API call specifying the job identifier may be made and a response providing information related to the job may result. Various API calls may be used to store data, retrieve data, obtain an inventory of stored data, and to obtain other information relating to stored data.

Data storage application programming interface

Techniques described and suggested herein include distributed deletion request processing and verification. For example, incident to migration of original data from a first data store to a second data store, verifications and confirmations related to removing the original data from the first data store may be performed so as to ensure the integrity of the original data represented on the second data store prior to removing the actual original data on the first data store. In some embodiments, the verifications and confirmations performed in connection with a deletion request may be apportioned to multiple entities, each of which may not fully trust the others. As a result, in some embodiments, a given deletion request may only be fulfilled if all of the entities involved in the verification process individually provide authorization to execute the deletion request.

Techniques for distributing verification checks incident to data deletion

A method and apparatus is disclosed for utilizing a validity map to indicate whether an overwriting data portion of user data is received. In the method and apparatus, a plurality of data portions of user data are received, whereby each data portion of the plurality of data portion may overwrite another data portion. A validity map may be utilized to indicate whether a data portion is an overwriting data portion. The validity map may be utilized for retrieving the data portion instead of the data portion being replaced.

Validity map-based tracking of user data updates

A data storage service redundantly stores data and keys used to encrypt the data. Data objects are encrypted with first cryptographic keys. The first cryptographic keys are encrypted by second cryptographic keys. The first cryptographic keys and second cryptographic keys are redundantly stored in a data storage system to enable access of the data objects, such as to respond to requests to retrieve the data objects. The second cryptographic keys may be encrypted by third keys and redundantly stored in the event access to a second cryptographic key is lost.

Redundant key management

Embodiments of the present disclosure are directed to, among other things, validating the integrity of received and/or stored data payloads. In some examples, a storage service may perform a first partitioning of a data object into first partitions based at least in part on a first operation. The storage service may also verify the data object, by utilizing a verification algorithm, to generate a first verification value. In some cases, the storage service may additionally perform a second partitioning of the data object into second partitions based at least in part on a second operation. The second partitions may be different from the first partitions. Additionally, the archival data storage service may verify the data object using the verification algorithm to generate a second verification value. Further, the storage service may determine whether the second verification value equals the first verification value.

Data storage integrity validation

Techniques described and suggested herein include implementations of caches and scalers to handle data storage requests, and storage event status requests associated with data storage requests, in a scalable fashion. For example, a data storage system, such as a data storage system implemented by a computing resource service provider in connection with providing an archival storage service or other data storage service, may be implemented to maintain a consistent response time and backend capability for incoming data storage requests, which may be a component of ensuring a consistent customer experience for customers of an associated service, with little or no regard to peaky or high data storage request rates observed by the implementing data storage system.

Techniques for adapting data storage systems for peaky usage models

A data storage system synchronously or asynchronously determines data storage usage policies in connection with incoming data storage requests so as to improve policy compliance and system efficiency. In some embodiments, a synchronous usage meter is used to make such policy decisions, and variously implement consistent hashing techniques to improve system availability and efficiency. In some embodiments, an asynchronous usage meter is used, and is implemented using an ordered queueing engine so as to improve the accuracy of policy decisions.

Techniques for usage metering and control in data storage systems

A cost-effective, durable and scalable archival data storage system is provided herein that allow customers to store, retrieve and delete archival data objects, among other operations. For data storage, in an embodiment, the system stores data in a transient data store and provides a data object identifier may be used by subsequent requests. For data retrieval, in an embodiment, the system creates a job corresponding to the data retrieval and provides a job identifier associated with the created job. Once the job is executed, data retrieved is provided in a transient data store to enable customer download. In various embodiments, jobs associated with storage, retrieval and deletion are scheduled and executed using various optimization techniques such as load balancing, batch processed and partitioning. Data is redundantly encoded and stored in self-describing storage entities increasing reliability while reducing storage costs. Data integrity is ensured by integrity checks along data paths.

Archival data storage system

Methods and systems are provided herein to allow efficient management of data flowing in and out of an archival data storage system. In an embodiment, storage entities keep very little state information in memory to provide higher throughput. Further, storage entities may send data in large chunks to facilitate high throughput. Techniques such as batching and coalescing may be used by various storage entities to provide efficiency.

Archival data flow management

Embodiments of the present disclosure are directed to, among other things, managing power of one or more data storage devices. In some examples, a storage service may obtain a schedule associated with enabling different storage devices at different times. The storage service may also identify a request of a batch of requests for accessing the storage devices. In some cases, the storage service may also determine which storage device to activate based at least in part on the schedule and/or the request. Further, the storage service may manage power of a storage device based at least in part on the determination of which storage device to activate.

Growing community of inventors

Seattle, WA, United States of America

Mark Christopher Seigle