Amazon Technologies, Inc.

Marc Stephen Olson

Marc John Brooker

James Michael Thompson

Danny Wei

Anthony Nicholas Liguori

Christopher Magee Greenwood

Pieter Kristian Brouwer

Benjamin Arthur Hawks

Norbert Paul Kusters

Surya Prakash Dhoolam

Mark Bradley Davis

Robert Michael Johnson

Nachiappan Arumugam

Sandeep Kumar

Kristina Kraemer Brenneman

John Luther Guthrie

Jerry Lin

Arpit Tripathi

David R Richardson

James T Pinkerton

Lalit Jain

Michael Thacker

Vijay Prasanth Rajavenkateswaran

Thomas A Volpe

Asif Khan

Andrew Kent Warfield

Madhuvanesh Parthasarathy

Brad Eugene Marshall

Jacob Taylor Wires

Asawaree Kalavade

John Robert Smiley

Avram Israel Blaszka

Mark Robinson

Arvind Chandrasekar

Valentin Gabriel Priescu

Paul C Reed

Brian Todd Burruss

Jeevan Shankar

Diwakar Gupta

Jianhua Fan

Mitchell Gannon Flaherty

Bhavesh Anil Doshi

Sriram Venugopal

Marc Levy

David Wachtfogel

Ankit Singh

Pavel Labovich

Vaibhav Sharma

Ganesh Viswanathan

Tarun Goyal

Valentin-Gabriel Priescu

Shweta Joshi

Manish Verma

Pavan Kumar Korlepara

Nobert Paul Kusters

Peter B Nigh

Mahesh Sreekandath

Stephen David Hildrey

Farhan Tanvir Ali

Rajiv Satish Nair

Systems and methods are provided for providing a consistent experience for users of cloud-based block storage volumes. While cloud storage aims to remove hardware considerations for an end user's experience, block storage performance can nevertheless vary according to the underlying hardware used to support a volume or the specific network location of that hardware. Embodiments of the present disclosure address that inconsistent performance by associating a volume with a performance profile that sets a target latency for the volume. A storage client can then monitor observed latency for the volume and inject synthetic latency into input/output operations for the volume as calculated via a proportional-integral-derivative algorithm, such that the observed latency matches the target within the performance profile. This enables the cloud provider to vary physical hardware or network configurations without effect on block storage performance from the point of view of an end user.

Providing consistent performance of block-storage volumes in cloud computing environments

Systems and methods are provided for handling file operations from a hosted computing instance via a secure compute layer. The secure compute layer is presented to the instance as a virtualized service device that is locally addressable by the instance. Software within the instance can submit file operations to the virtualized service device, which the secure compute layer can translate into calls to a network-accessible storage service. Results from the calls can then be passed back to the instance through the virtualized service device. As a result, the instance can communicate with a variety of different network services, without itself implementing network communications for those services.

Offloading of remote service interactions to virtualized service devices

Systems and methods are provided for implementing a multi-service file system for a hosted computing instance via a locally-addressable secure compute layer. Software within the instance can submit file operations to the secure compute layer, which the secure compute layer can translate into calls to one or more network-accessible storage services. To provide a multi-service file system, the secure compute layer can obtain mapping data mapping file system objects within the virtualized file system to different network-accessible storage services. On receiving a file operation, the secure compute layer can determine one or more network-accessible storage services corresponding to the file operation, and submit appropriate calls to the one or more network-accessible storage services. By varying the calls for file operations, various functionalities, such as data backup, write staging, read caching, and failover can be implemented independent of both operation of the hosted computing device and the network-accessible storage services.

Pseudo-local multi-service enabled file systems using a locally-addressable secure compute layer

Systems and methods for efficient and secure management of encrypted 'snapshots' for a remote provider substrate extension ('PSE') of a cloud provider network substrate are provided. The PSE may request and obtain a snapshot from the cloud provider network substrate, restore a volume from the snapshot, make changes to data in the restored volume, and/or initiate the creation and storage of a new snapshot that includes incremental updates to the original snapshot to reflect the changes made to data in the volume. An encrypted snapshot stored within the cloud provider network substrate may be decrypted using a cloud provider key designed for internal use only, and then re-encrypted using a PSE-specific key before providing the snapshot to the PSE, thereby avoiding the sharing of the cloud provider internal use only key outside the cloud provider network substrate.

Snapshot management across cloud provider network extension security boundaries

A first block storage server virtual machine to host a first volume using one or more storage devices of a computer system is executed by the computer system. A second virtual machine having access to a virtual block storage device is executed by the computer system. A block storage client is executed by the computer system. A first block storage operation is received by the block storage client from the second virtual machine, the first block storage operation to perform on the virtual block storage device. A message is sent by the block storage client to the first block storage server virtual machine to cause the first block storage server virtual machine to perform the block storage operation with the first volume.

Virtualized block storage servers in cloud provider substrate extension

One or more configuration parameters for an object gateway instance are received at an interface to a provider network, the parameters including an identifier of a first object store of the provider network for which to cache objects in a first object cache of the object gateway instance and an indication of a data transfer mode that controls when objects written to the first object cache are written to the first object store. The one or more configuration parameters are stored in a data store of the provider network and sent to the object gateway instance. A read request that includes the identifier of the first object store and a first object identifier is received from the object gateway instance, and a first object associated with the first object identifier and stored in the first object store is sent to the object gateway instance.

Data caching in provider network substrate extensions

A data storage manager may manage storage locations for blocks of a storage volume. The blocks of the storage volume may be assigned to a logical volume exposed to a computing instance supported by a host. Furthermore, the data storage manager may also generate and maintain a set of rules that specify the locations of blocks of the storage volume, and provides the set of rules to the host. The set of rules may be included in a data structure enabling the host to access the blocks based at least in part on the information included in the set of rules.

Enabling host computer systems to access logical volumes by dynamic updates to data structure rules

Security can be provided for data stored using resources that are deployed in an environment managed by a third party. Physical and logical detection mechanisms can be used to monitor various security aspects, and the resulting security data can be used to identify potential threats to these resources. In some embodiments, suspicious activity can cause resources such as data servers to be automatically and remotely rebooted such that keys stored in volatile memory on those data servers will be lost from those servers, such that an attacker will be unable to decrypt data stored on those servers. Once a determination of safety is made, the keys can be provided to the respective data servers such that data operations can resume.

Managing security in isolated network environments

Systems and methods are provided for providing a consistent experience for users of cloud-based block storage volumes. While cloud storage aims to remove hardware considerations for an end user's experience, block storage performance can nevertheless vary according to hardware configuration, such as the specific model of underlying hardware used to support a volume or the specific network location of that hardware. Embodiments of the present disclosure address that inconsistent performance by associating a volume with a performance profile, which can represent a desired distribution of performance for the volume. Each read or write can be compared to the profile, and may be selectively delayed to match the desired performance. This enables the cloud provider to vary physical hardware or network configurations without effect on block storage performance from the point of view of an end user.

Delaying result of I/O operation based on target completion time

A first one or more messages is received, the one or more messages including a request for a storage expansion device for an extension of a provider network, an identifier of the extension of the provider network, and a set of one or more identifiers associated with objects to load to the storage expansion device. For each identifier in the set, an object associated with the identifier is copied from an object store of the provider network to the storage expansion device. A shipment of the storage expansion device to a specified location is initiated. The extension of the provider network is caused to launch an instance to communicate with the storage expansion device upon connection of the storage expansion device to the extension of the provider network.

Storage expansion devices for provider network substrate extensions

A user can set or modify operational parameters of a data volume stored on a network-accessible storage device in a data center. For example, the user may be provided access to a data volume and may request a modification to the operational parameters of the data volume. Instead of modifying the existing data volume, the data center can provision a new data volume and migrate data stored on the existing data volume to the new data volume. While the data migration takes place, the existing data volume may block input/output (I/O) requests and the new data volume may handle such requests instead. Once the data migration is complete, the data center may deallocate the data blocks of the existing data volume such that the data blocks can be reused by other data volumes.

Network-accessible data volume modification

A distributed system may implement identifying correlated workloads for resource allocation. Resource data for resources hosted at resource hosts in a distributed system may be analyzed to determine behavioral similarities. Historical behavior data or resource configuration data, for instance, may be compared between resources. Behaviors between resources may be identified as correlated according to the determined behavioral similarities. An allocation of one or more resource hosts in the distributed system may be made for a resource based on the behaviors identified as correlated. For instance, resources may be migrated from a current resource host to another resource host, new resources may be placed at a resource host, or resources may be reconfigured into different resources. Machine learning techniques may be implemented to refine techniques for identifying correlated behaviors.

Identifying correlated resource behaviors for resource allocation

Techniques are described for using in-band communication channels to exchange state information between components of a distributed storage environment, including between client computing devices and storage servers hosting network-connected block storage volumes. The exchange of the state information can be used, for example, to inform client computing devices of relevant events involving one or more storage volumes attached to compute instances (for example, virtual machines (VMs)) running on the client computing devices, involving one or more failover servers storing backup copies of one or more storage volumes, or involving any other relevant system components. The exchange of such state information in a distributed storage environment enables client computing devices, storage servers, and other system components to act efficiently in response to a variety of system events thereby mitigating input/output (I/O) latency increases and other issues that can arise when system components lack access to such information.

Communicating state information in a distributed storage environment

Memory is identified for a first set of histogram buckets for a metric with bucket value ranges distributed according to a first function, and a second set of histogram buckets with bucket value ranges distributed according to a second function. The second set of buckets overlaps with a metric value range targeted for enhanced granularity analysis. After the histogram is updated in response to obtaining metric values, a representation of at least some of the buckets is sent to a destination.

Tunable-granularity multi-level histograms for efficient computer system metric analysis

Methods, systems, and computer-readable media for program code allocation based on processor features are disclosed. Analysis of program code is performed using static analysis and/or runtime analysis. The analysis determines one or more processor features invoked by the program code. One or more program execution servers are selected from a plurality of program execution servers based at least in part on the one or more processor features invoked by the program code. One or more selected program execution servers comprise the one or more processor features invoked by the program code, and an additional one or more of the program execution servers lack the one or more processor features invoked by the program code. The program code is executed using the one or more selected program execution servers.

Program code allocation based on processor features

Growing community of inventors

Bellevue, WA, United States of America

Marc Stephen Olson