Amazon Technologies, Inc.

Microsoft Technology Licensing, LLC

Facebook, Inc.

Meta Platforms, Inc.

Kestutis Patiejunas

Colin Laird Lazier

Mark Christopher Seigle

James Christopher Sorenson

Christian L Claiborn

Pradeep Vincent

Bryan James Donlan

David Carl Salyers

Ankur Khetrapal

Eric W Nace

Giedrius Zizys

Dario Bazan Bejarano

Todd R Manion

Giovanni Coglitore

Narsing Krishna Vijayrao

Ryan Yonghee Kim

Claire E Suver Weir

James Robert Hamilton

Sing Bing Kang

Paul David Franklin

Charles Lawrence Zitnick, Iii

Simon Winder

David Eraker

Adam G Kirk

Claire Elizabeth Suver

Junfeng Zhang

Patrick John Sweeney

Alyssa H Henry

Yaron Eshet

Weixiang Sun

Cong Chen

Gian-Paolo Primo Musumeci

Claire E Suver

James Christopher Sorenson Iii

An application programming interface for a data storage service provides a convenient mechanism for clients of the data storage service to access its various capabilities. An API call may be made to initiate a job and in response a job identifier may be provided. A separate API call specifying the job identifier may be made and a response providing information related to the job may result. Various API calls may be used to store data, retrieve data, obtain an inventory of stored data, and to obtain other information relating to stored data.

Data storage application programming interface

According to examples, a system for rate-based load balancing may include a processor and a memory storing instructions. The processor may, through execution of the instructions, cause the system to receive a request for processing. The system may further identify a target server to transmit the request using a rate-based load balancing technique. In some examples, the rate-based load balancing technique may include: selecting a server, from a plurality of servers, as a potential target; receiving a readiness indicator for the selected server; and designating the selected server as the target server based on the readiness indicator. The system may transmit the request to the target server for processing.

Systems and methods for rate-based load balancing

Methods and apparatus for supporting cached volumes at storage gateways are disclosed. A storage gateway appliance is configured to cache at least a portion of a storage object of a remote storage service at local storage devices. In response to a client's write request, directed to at least a portion of a data chunk of the storage object, the appliance stores a data modification indicated in the write request at a storage device, and asynchronously uploads the modification to the storage service. In response to a client's read request, directed to a different portion of the data chunk, the appliance downloads the requested data from the storage service to the storage device, and provides the requested data to the client.

Cached volumes at storage gateways

Embodiments of the present disclosure are directed to, among other things, validating the integrity of received and/or stored data payloads. In some examples, a storage service may perform a first partitioning of a data object into first partitions based at least in part on a first operation. The storage service may also verify the data object, by utilizing a verification algorithm, to generate a first verification value. In some cases, the storage service may additionally perform a second partitioning of the data object into second partitions based at least in part on a second operation. The second partitions may be different from the first partitions. Additionally, the archival data storage service may verify the data object using the verification algorithm to generate a second verification value. Further, the storage service may determine whether the second verification value equals the first verification value.

Data storage integrity validation

Techniques for implementing a data management scheme for optimizing data storage are described herein. A deletion quantity or other metric relating to deletions of data within a first storage zone are tracked. Upon detection that the tracked deletion metric meets certain criteria, the data within the first storage zone are moved to a second storage zone. A verification of the data to be moved is also performed, and if such verification indicates that at least a portion of the data is corrupted, routines repairing and/or restoring at least the corrupted portion are initiated.

Data storage management for sequentially written media

A system for storing data includes a rack, one or more data storage drive assemblies coupled to the rack, and a data control module coupled to the rack. The data storage drive assemblies include one or more drive mechanical modules configured to store data and one or more drive control modules coupled to the drive mechanical modules. The drive control modules control mechanical operations in the drive mechanical modules. The drive mechanical modules and the associated drive control modules are separable from one another without removing the other module from the at least one data storage drive assembly.

Hard disk drive assembly with field-separable mechanical module and drive control

Methods and systems are described herein to provide efficient data retrieval in a data storage system. Specifically, in cases where users of a data storage system are not overly sensitive to data retrieval time, such as the case for backup and archival data storage systems, random read requests may be fulfilled as part of sequential reads to reduce I/O operations. A data storage system may be divided into data storage zones. Sequential reads may be performed for data stored in those data storage zones with pending data retrieval requests. Data retrieval requests may be fulfilled based at least in part on the sequentially-read data.

Efficient data retrieval based on random reads

Embodiments of the present disclosure are directed to, among other things, managing space allocation of one or more data storage devices. In some examples, a storage service may identify at least one computer-readable storage device. The computer-readable storage device may, in some examples, have availability level that corresponds to a size of data to be stored. The storage service may also identify a target storage device based at least in part on a capacity of a subset of the storage devices. In some cases, this may be performed for one or more of the devices of the subset. Further, the storage service may store the data in the identified target storage device.

Data storage space management

Techniques for implementing a log-based storage scheme upon data storage devices are described herein. A data storage device is initialized by writing an identifying record. For each portion of data to be written to the drive, a first record including information regarding the anticipated nature of the portion of data is written prior to the data. The data is then written as a second record that includes at least the raw data as well as integrity verification information. A third record is stored following the second record, and includes an accounting and/or index of the data successfully written in the second record. On sequentially written devices, the information in the first stored record may be used to locate the third record, which in turn may be used to record data in the second record as well as the location of a first record of another portion of data.

Log-based data storage on sequentially written media

A cost-effective, durable and scalable archival data storage system is provided herein that allow customers to store, retrieve and delete archival data objects, among other operations. For data storage, in an embodiment, the system stores data in a transient data store and provides a data object identifier may be used by subsequent requests. For data retrieval, in an embodiment, the system creates a job corresponding to the data retrieval and provides a job identifier associated with the created job. Once the job is executed, data retrieved is provided in a transient data store to enable customer download. In various embodiments, jobs associated with storage, retrieval and deletion are scheduled and executed using various optimization techniques such as load balancing, batch processed and partitioning. Data is redundantly encoded and stored in self-describing storage entities increasing reliability while reducing storage costs. Data integrity is ensured by integrity checks along data paths.

Archival data storage system

Embodiments of the present disclosure are directed to, among other things, managing inventory indexing of one or more data storage devices. In some examples, a storage service may store an index associated with archived data. Additionally, the storage service may receive information associated with an operation performed on the archived data. The storage service may also partition the received information into subsets corresponding to an identifier. In some cases, the identifier may be received with or otherwise be part of the received information. The storage service may also retrieve at least a portion of the index that corresponds to the subset. Further, the storage service may update the retrieved portion of the index with at least part of the received information. The updating may be based at least in part on the subsets.

Data storage inventory indexing

Various embodiments ('systems') are described for transferring data from a primary storage (e.g., magnetic disk drives, solid state drives, etc.) to an optical cold storage rack. The optical cold storage rack may include many physical optical storage disks, but a much smaller number of burners and readers (e.g., optical disk drives). When data is to be transferred to the optical cold storage rack, the system may generate a plan for performing the transfer. 'Migration worker' components may then implement the plan and may be exclusively dedicated to implementing such plans. In various embodiments, the plan may specify how large data file &#x201c;aggregates&#x201d; (collections of portions of one or more data files) are to be distributed across optical disks (&#x201c;disks&#x201d;) to improve throughput during subsequent reading operations from the optical cold storage rack. The plan may also anticipate the relation between the limited number of burners/readers and the overall optical cold storage rack disk capacity.

Optical cold storage with encoding and placement of data chunks on different optical disks

Growing community of inventors

Sammamish, WA, United States of America

Kestutis Patiejunas