Amazon Technologies, Inc.

Grant A M McAlister

Swaminathan Sivasubramanian

Rajesh Sudhakar Sheth

Stefano Stefani

Timothy Andrew Rath

Rande Anthony Blackman

Raymond S Bradford

Wei Xiao

Jakub Kulesza

Chiranjeeb Buragohain

Luis Felipe Cabrera

James Robert Hamilton

Brandon William Porter

Paul David Franklin

Gokul Soundararajan

Jeromey Russell Goetz

Jeff Bradley Beal

Alan Stuart Robbins

Joseph L Ellsworth

Milovan Milovanovic

James Horsley

A system that implements a scaleable data storage service may maintain tables in a data store on behalf of storage service clients. The service may maintain data in partitions stored on respective computing nodes in the system. The service may support multiple throughput models, including a committed throughput model and a best effort throughput model. A service request to create a table may specify that requests directed to the table should be serviced under a committed throughput model and may specify the committed throughput level in terms of logical service request units. The service may reserve low-latency storage and other resources sufficient to meet the specified committed throughput level. A client/user may request a modification to the committed throughput level in anticipation of workload changes, such as an increase or decrease in traffic or data volume. In response, the system may increase or decrease the resources reserved for the table.

System and method for providing a committed throughput level in a data store

Access level and security group information can be updated for a data instance without having to take down or recycle the instance. A data instance created in a data environment will have at least one default security group. Permissions can be applied to the default security group to limit access via the data environment. A control security group can be created in a control environment and associated with the default security group. Permissions can be applied and updated with respect to the control security group without modifying the default security group, such that the data instance does not need to be recycled or otherwise made unavailable. Requests to perform actions with respect to the control security groups are made via the control environment, while allowing native access to the data via the data environment.

Managing security groups for data instances

System and methods for managing versions of partition maps in a distributed data store are disclosed. A relational database may use partition maps to indicate the location of data in a particular partition, or in a replica thereof. These partition maps may be versioned using a two-stage mechanism in which partition map updates are communicated to all nodes, or to affected nodes only, and in which operation may continue without waiting for acknowledgment of their application to each node. The system may handle minor partition changes differently than major partition changes. For example, if a replica of a partition is promoted to be a primary copy due to hardware or network failures, partition map changes may only be propagated to nodes in the same replica group. For major partition changes (e.g., splits, moves, mergers, added partitions, or size change), a new partition map may be made available to all nodes.

Versioning of database partition maps

Various embodiments of a system and method for tracking service requests are described. Embodiments may include call tree generation logic configured to receive multiple request identifiers associated with a respective one of multiple service requests. Each given request identifier may include an origin identifier, a depth value, and a request stack comprising one or more interaction identifiers. The call tree generation logic may also be configured to, based on multiple request identifiers that each include an origin identifier associated with a particular root request, generating a data structure that specifies a hierarchy of services called to fulfill that particular root request. Based on one or more of the interaction identifiers and one or more of the depth values, the generated data structure may specify for each given service of the hierarchy: a parent service that called the given service, and one or more child services called by the given service.

System and method for tracking service results

A system that implements a scaleable data storage service may maintain tables in a non-relational data store on behalf of service clients. Each table may include multiple items. Each item may include one or more attributes, each containing a name-value pair. The system may provide an API through which clients can query tables maintained by the service. Items may be partitioned and indexed in a table according to a simple or composite primary key contained in all items in the table. A composite primary key may include a hash key attribute, and a range key attribute. The range key attribute may be usable to order items having the same hash key attribute value, and to partition them dependent on a range of range key attribute values. A query request may specify a logical or mathematical expression dependent on range key attribute values and may be directed to multiple partitions.

System and method for partitioning and indexing table data using a composite primary key

Aspects of a data environment, such as the creation, provisioning, and management of data stores and instances, are managed using a separate control environment. A user can call into an externally-facing interface of the control environment, the call being analyzed to determine actions to be performed in the data environment. A monitoring component of the control plane also can periodically communicate with the data environment to determine any necessary actions to be performed, such as to recover from faults or events in the data environment. A workflow can be instantiated that includes tasks necessary to perform the action. For each task, state information can be passed to a component in the data environment operable to perform the task, until all tasks for an action are completed. Data in the data environment can be accessed directly using an externally-facing interface of the data environment, without accessing the control plane.

Control service for relational data management

A system that implements a scalable data storage service may maintain tables in a non-relational data store on behalf of clients. The system may provide a Web services interface through which service requests are received, and an API usable to request that a table be created, deleted, or described; that an item be stored, retrieved, deleted, or its attributes modified; or that a table be queried (or scanned) with filtered items and/or their attributes returned. An asynchronous workflow may be invoked to create or delete a table. Items stored in tables may be partitioned and indexed using a simple or composite primary key. The system may not impose pre-defined limits on table size, and may employ a flexible schema. The service may provide a best-effort or committed throughput model. The system may automatically scale and/or re-partition tables in response to detecting workload changes, node failures, or other conditions or anomalies.

System and method for implementing a scalable data storage service

Systems and methods for providing resource isolation in a shared computing environment using reinforcement learning (RL) techniques are disclosed. A resource isolation mechanism may be applied in a shared storage system, or database service, that limits the resource utilization of each namespace to its specified allocation. For example, the resource isolation mechanism may be used to limit the I/O utilization of database applications in a shared computing system (e.g., a system supporting a database service) to a specified limit. In such embodiments, RL techniques may be applied to the system to automatically control the rate of queries made by an application. RL techniques, such as those based on the State-Action-Reward-State-Action (SARSA) method may be effective in controlling the I/O utilization of database applications for different workloads. RL techniques may be applied globally by the service, or may be applied to particular subscribers, applications, shared resources, namespaces, or query types.

Resource isolation through reinforcement learning

System and methods for providing a Scalable Relational Database Service (SRDBS) are disclosed. SRDBS may allow a subscriber to lease/rent a virtual relational database (a.k.a. namespace) with a specified allocation of disk space, CPU resources and/or I/O resources. The namespace may resemble a physical RDBMS, and may allow client applications to perform SQL queries. SRDBS may partition a subscriber's database across multiple database servers automatically and may replicate each partition automatically. SRDBS may provide automatic routing of client queries to the appropriate database server holding the required data. The routing process may mask temporary data outages due to database replica failures, network failures, and/or repartitioning operations. SRDBS may employ partition maps to convey the location of data to client-side routing functions. These partition maps may use a multi-versioning scheme. SRDBS may provide a resource isolation mechanism that limits the resource utilization of each namespace to its specified allocation.

Scalable relational database service

Aspects of a data environment, such as the monitoring, backup, and recovery of data stores and instances, are managed using a separate control environment. A monitoring component of the control environment can periodically communicate with the data environment to determine any necessary actions to be performed, such as to recover from faults or events for a data instance in the data environment. A workflow can be instantiated that includes tasks necessary to perform a recovery action. Tasks of the workflow can instantiate a new data instance and, using an identifier for the original data instance, access a snapshot for the data instance and copy the data to the new instance. Log files also can be accessed and applied to the new data instance to cause the new instance to represent the state of the original data instance at a particular point in time.

Growing community of inventors

Seattle, WA, United States of America

Grant A M McAlister