Amazon Technologies, Inc.

Eric Neilsen

Frank Charles Paterra

Firat Basarir

Andrew Portnoy

Paul David Franklin

Xiao Li

Shuo Yang

Vaibhav Tyagi

Apoorv Birthare

Mahadeva Navali Basavaraj

Jeffry Strunk

Ho Ming Li

Elodie Wun

Technologies are disclosed for migrating large amounts of data objects, e.g., billions of data objects, from a distributed data store to another data store or computing device. A method includes generating an inventory of the data objects in the distributed data store and dividing the inventory into multiple listings of data objects. Each data object is identified in only one corresponding listing. Manifests are generated based upon the listings. Each manifest is assigned to a corresponding portable storage device of a plurality of portable storage devices, and the data objects and listings are transferred, based at least in part, upon the manifests to the plurality of portable storage devices. The inventory of the data objects and the plurality of portable storage devices are provided to an entity associated with the another data store or computing device. The data objects are transferred from the plurality of portable storage devices to the another data store or computing device using the inventory.

Migrating data objects from a distributed data store to a different data store using portable storage devices

A data storage system implements aggregation, bifurcation, and/or reduction techniques to improve the efficiency of processing data storage requests. Data storage requests and/or their associated payloads may be aggregated based on one or more parameters. Data to be the stored and the associated commands may be separated so as to optimize a system's throughput and latency for each. Furthermore, extraneous commands and requests may be reduced or eliminated based on heuristics associated with the requests and the data.

Client-based batching of data payload

Data payload clustering for data storage systems

A device such as a network-attachable data transfer device may be configured to operate in a cluster to coordinate the storage of data. A cluster manifest may include data and/or metadata related to devices of the cluster. In some cases, a cluster manifest may include, for each of one or more devices of a cluster: an encrypted payload; an identifier associated with the particular device; and an encrypted data entry. The encrypted payload may encode a cryptographic payload key used in part to perform storage operations. The encrypted data entry may include one or more encrypted partitions that are each decryptable by a different security module of the device. The decrypted partitions may be assembled to form a cryptographic key that may decrypt the encrypted payload. The cluster manifest may be provided to a locked device of a cluster as part of a process for unlocking the locked device.

Cluster of network-attachable storage devices with cluster manifest

A processing device receives a request to identify whether a file is a complete file. The processing device determines a first size of the file. The processing device determines whether the first size satisfies a size criterion. The processing device determines that the file is a complete file responsive to the determination that the first size satisfies the size criterion.

File size as an indicator of file properties

A data storage system implements aggregation, bifurcation, and/or reduction techniques to improve the efficiency of processing data storage requests. Data storage requests and/or their associated payloads may be aggregated based on one or more parameters. Data to be stored and the associated commands may be separated so as to optimize a system's throughput and latency for each. Furthermore, extraneous commands and requests may be reduced or eliminated based on heuristics associated with the requests and the data.

Command aggregation for data storage operations

Data payload aggregation for data storage systems

Command parallelization for data storage systems

Contextual optimization for data storage systems

Growing community of inventors

Lynnwood, WA, United States of America

Eric Neilsen