Amazon Technologies, Inc.

Kiran-Kumar Muniswamy-Reddy

Wei Xiao

Miguel Mascarenhas Filipe

Bjorn Patrick Swift

Yijun Lu

Stuart Henry Seelye Marshall

Wells Lin

Timothy Andrew Rath

Danny Wei

Stefano Stefani

James Robert Hamilton

Christopher Magee Greenwood

Igor A Kostic

Divya Ashok Kumar Jain

Colin Williams

William Zaharchuk

Barry Bailey Hunter, Jr

Rahul Upadhyaya

Romain Benoit Seguy

Jai Vasanth

Allan Henry Vermeulen

David Alan Lutz

Timothy Daniel Cole

Mukul Vijay Karnik

Vishal Parakh

Rande Anthony Blackman

Antoun Joubran Kanawati

Ajay Gopalakrishnan

Omer Baluch

Si Yin

Norbert Paul Kusters

David Craig Yanacek

Maximiliano Maccanti

Xianglong Huang

Julio Delgado Mangas

Jian Wang

Akshat Vig

Somasundaram Perianayagam

Qiang Liu

Adam Douglas Morley

Parikshit Shivajirao Pol

Stuart Henry Marshall

Clarence Wing Yin Ng

Jiandan Zheng

Ying Lin

A resource management service implements techniques for provisioning a process with computing resources sufficient to process a query. A query is received and computing system resources sufficient to process the query are provisioned. A response to the query is generated by running the process with the provisioned computing system resources.

Resource management service

The present disclosure generally relates to creating virtualized block storage devices whose data is replicated across isolated computing systems to lower risk of data loss even in wide-scale events, such as natural disasters. The virtualized device can include at least two volumes, each of which is implemented in a distinct computing system. Each volume can be encrypted with a distinct key, and an encryption service can operate to transform data 'in-flight' on the replication path between the volumes, reencrypting data according to the key appropriate for each volume.

Encrypted cross-zone replication for cross-zone replicated block storage devices

The present disclosure generally relates to creating virtualized block storage devices whose data is replicated across isolated computing systems to lower risk of data loss even in wide-scale events, such as natural disasters. The virtualized device can include at least two volumes, each of which is implemented in a distinct computing system. Each volume can be implemented by at least two computing devices, a first of which is configured as a primary device to which reads from and writes to the volume are directed. Of the two volumes, one can be indicated as primary, indicating authority to accept reads to and writes from the virtualized device. A primary device of the primary volume, on obtaining a write to the volume, can replicate the write to both a secondary device of a primary volume and to the secondary volume.

Cross-zone replicated block storage devices

Generally described, aspects of the present application correspond to maintaining a message stream for a network-based data store, which stream includes messages reflecting modifications to the data store. Messages within the stream may be used to revert a state of the data store to a prior point in time reflected within the messages of the stream, such as by 'rewinding' operations on the data store by use of the messages within the stream. Messages in the stream may further be used to asynchronously update a replica of the data store.

Stream-based logging for distributed storage systems

A data analytics system may receive query definitions from which relationships between datasets may be identified. The query definitions may be analyzed to determine estimated costs and frequencies of combining a first and second dataset. Based on the cost and frequency, a combined dataset may be generated to by joining data from the first and second datasets. The combined dataset may be stored. Queries that comprise instructions to combine the first and second datasets may be processed by instead accessing the combined dataset.

Analysis engine data intake

Updates for clients accessing a data volume across fault tolerance zones may be replicated. Requests to write or read a block of a logical volume with replicas located in different fault tolerance zones may be sent to the different replicas. Responses for the different requests may be evaluated to determine whether quorum is satisfied for the write or read of the block of the logical volume. For writes that satisfy quorum, a request to commit the write may be sent to the replicas of the logical volume.

Replicating data volume updates from clients accessing the data volume across fault tolerance zones

The present disclosure generally relates to creating virtualized block storage devices whose data is replicated across isolated computing systems to lower risk of data loss even in wide-scale events, such as natural disasters. The virtualized device can include at least two volumes, each of which is implemented in a distinct computing system. In the case of a failed volume, a new volume can be created and populated with data from the surviving volume. During population, new writes can continue to be replicated to the new volume. The population process can write data from the surviving volume to the new volume 'under' new writes, such that the population process does not overwrite data included in the new writes.

Failover for failed secondary in cross-zone replicated block storage devices

The present disclosure generally relates to creating virtualized block storage devices whose data is replicated across isolated computing systems to lower risk of data loss even in wide-scale events, such as natural disasters. The virtualized device can include at least two volumes, each of which is implemented in a distinct computing system. Each volume can be implemented by at least two computing devices, a first of which is configured as a primary device to which reads from and writes to the volume are directed. To ensure consistency in the distributed device, a multi-tier authority service is implemented, in which a cross-computing system authority service designates a volume as having authority to accept writes to the virtualized device, and in which a second tier authority service designates a computing device as having authority to accept writes to the volume.

Hierarchical authority store for cross-zone replicated block storage devices

Generally described, one or more aspects of the present application correspond to a highly distributed replica of a volume stored in a networked elastic computing environment. First and second replicas of the volume can be synchronously replicated, and some implementations of the tertiary replica can be asynchronously replicated. The highly distributed nature of the tertiary replica supports parallel data transfer of the data of the volume, resulting in faster creation of backups and new copies of the volume.

Rapid volume backup generation from distributed replica

A snapshot object or other type of object may be stored in a first storage system and may be accelerated in another storage system, wherein an accelerated snapshot or other type of object can be used to populate volumes in the other storage system with data more rapidly than a non-accelerated snapshot or other type of object. The accelerated snapshot or other object may be implemented using an intermediate volume implemented in the other storage system that is populated with data from the snapshot object or the other object stored in the first storage system.

Accelerated volumes

Methods and apparatus for token-based admission control for replicated writes are disclosed. Data objects are divided into partitions, and corresponding to each partition, at least a master replica and a slave replica are stored. A determination as to whether to accept a write request directed to the partition is made based at least in part on one or more of (a) available throughput capacity at the master replica, and (b) an indication, obtained using a token-based protocol, of available throughput capacity at the slave replica. If the write request is accepted, one or more data modification operations are initiated.

Token-based admission control for replicated writes

Generally described, aspects of the present application correspond to enabling rapid duplication of data within a data volume hosted on a network storage system. The network storage system can maintain a highly distributed replica of the data volume, designated for duplication of data within the volume and separate from one or more other replicas designated for handling modifications to the data volume. By providing increased parallelization, the highly distributed replica can facilitate rapid duplication of the volume. When a sufficiently large request to duplicate the data volume is received, the system can create additional duplicate portions of the volume to further increase parallelization. For example, a partition of the highly distributed replica may be repeatedly duplicated to create a large number of intermediary duplicate partitions. The intermediary duplicate partitions can then be used to service the duplication request rapidly, due to increased parallelism.

Intermediary duplication to facilitate copy requests in distributed storage systems

The present disclosure generally relates to creating virtualized block storage devices whose data is replicated across isolated computing systems to lower risk of data loss even in wide-scale events, such as natural disasters. The virtualized device can include at least two volumes, each of which is implemented in a distinct computing system. Due to separation between volumes, replication lag may occur, in which data persisted to a first volume is not immediately persisted to a second volume. Such lag can increase a potential for data loss in the event that the first volume fails. Embodiments of the present disclosure relate to managing data loss risk by determining an expected maximum difference between the data stored at the two volumes, in a manner that does not require decrypting the data written to the volumes or perfect knowledge of the state of the distributed system at a single point.

Approximating replication lag in cross-zone replicated block storage devices

Providers of web services and other types of software as a service may be subject to service-level agreements requiring that response times be within a defined range. For efficiency, multiple services may be hosted on the same set of computing nodes, which may jeopardize adherence to service-level agreements. A control system may involve classifying service requests and determining desired values for measurements such as latency. An error value may be calculated based on the difference between measured and desired values. A controller may adjust a rate of capacity utilization for the computing nodes based on the current error, a history of past errors, and a prediction of future errors.

Real-time monitoring of IO load and latency

Growing community of inventors

Seattle, WA, United States of America

Kiran-Kumar Muniswamy-Reddy