Amazon Technologies, Inc.

Timothy Andrew Rath

David Alan Lutz

Wei Xiao

Jakub Kulesza

Stefano Stefani

Swaminathan Sivasubramanian

Rande Anthony Blackman

Maximiliano Maccanti

Stuart Henry Seelye Marshall

Santosh Kalki

Srinivasan Sundar Raghavan

Akshat Vig

Raymond S Bradford

Allan Henry Vermeulen

Mukul Vijay Karnik

Miguel Mascarenhas Filipe

Kiran-Kumar Muniswamy-Reddy

Amol Devgan

Clarence Wing Yin Ng

James Robert Hamilton

Rama Krishna Sandeep Pokkunuri

William Alexander Stevenson

Rajaprabhu Thiruchi Loganathan

Srivaths Badrinath Copparam

Mustafa Ozan Ozen

Luis Felipe Cabrera

Grant Alexander Macdonald McAlister

David Craig Yanacek

Chiranjeeb Buragohain

Bjorn Patrick Swift

Yijun Lu

Michael T Helmick

Lonnie J Princehouse

Chiranjeeb Buragahain

Timothy Daniel Cole

Gaurav Saxena

Jai Vasanth

Sandeep Bhatia

Grant A M McAlister

Jiandan Zheng

Anurag Windlass Gupta

Gaurav Gupta

Yan Valerie Leshinsky

Dumanshu Goyal

Adam Douglas Morley

Jian Wang

Atilim Cetin

Janani Narayanan

Somasundaram Perianayagam

Omer Ahmed Zaki

Xianglong Huang

Barry Bailey Hunter, Jr

Kanishka Chaturvedi

Eric Coll

Grant A Mcalister

Parikshit S Pol

Qiang Liu

Vishal Parakh

Antoun Joubran Kanawati

Medhavi Dhawan

Ajay Gopalakrishnan

Sudipto Das

Parikshit Shivajirao Pol

Karthik Gurumoorthy Subramanya Bharathy

Jared Scott Lundell

Zhong Ren

Grant Alexander MacDonald McAlister

Omer Baluch

Si Yin

Julio Delgado Mangas

Bhupinder Singh Sidana

Nirmesh Khandelwal

Ying Lin

Vincent Anthony Brancato

Kiran-Kumar Muniswamy Reddy

Avinash Goutham Reddy Aluguvelly

Zhihao Li

Rolf Petter Skog

Nilesh Shahdadpuri

Maximliano Maccanti

Stuart Henry Marshall

Clarence Wing Ng

Anurang Windlass Gupta

Grant A M Mcalister

A system that implements a scaleable data storage service may maintain tables in a data store on behalf of storage service clients. The service may maintain data in partitions stored on respective computing nodes in the system. The service may support multiple throughput models, including a committed throughput model and a best effort throughput model. A service request to create a table may specify that requests directed to the table should be serviced under a committed throughput model and may specify the committed throughput level in terms of logical service request units. The service may reserve low-latency storage and other resources sufficient to meet the specified committed throughput level. A client/user may request a modification to the committed throughput level in anticipation of workload changes, such as an increase or decrease in traffic or data volume. In response, the system may increase or decrease the resources reserved for the table.

System and method for providing a committed throughput level in a data store

A system that implements a scalable data storage service may maintain tables in a non-relational data store on behalf of clients. The system may provide a Web services interface through which service requests are received, and an API usable to request that a table be created, deleted, or described; that an item be stored, retrieved, deleted, or its attributes modified; or that a table be queried (or scanned) with filtered items and/or their attributes returned. An asynchronous workflow may be invoked to create or delete a table. Items stored in tables may be partitioned and indexed using a simple or composite primary key. The system may not impose pre-defined limits on table size, and may employ a flexible schema. The service may provide a best-effort or committed throughput model. The system may automatically scale and/or re-partition tables in response to detecting workload changes, node failures, or other conditions or anomalies.

System and method for implementing a scalable data storage service

A system that implements a data storage service may store data on behalf of storage service clients. The system may maintain data in multiple replicas of various partitions that are stored on respective computing nodes in the system. The system may employ a single master failover protocol, usable when a replica attempts to become the master replica for a replica group of which it is a member. Attempting to become the master replica may include acquiring a lock associated with the replica group, and gathering state information from the other replicas in the group. The state information may indicate whether another replica supports the attempt (in which case it is included in a failover quorum) or stores more recent data or metadata than the replica attempting to become the master (in which case synchronization may be required). If the failover quorum includes enough replicas, the replica may become the master.

System and method for data replication using a single master failover protocol

Techniques for performing a scratch space flush of cold tier upsert data into cold tier storage while allowing continued hot tier ingestion are described. Data points to be written into a time series database having a hot tier and a cold tier may be designated to be written directly into the cold tier, bypassing the hot tier that typically stores new data points that eventually are aged out into the cold tier. New data points for insertion directly into the cold tier may be stored to a scratch space storage and moved into the cold tier according to a non-blocking protocol that allows for continued hot tier ingestion without significant service disruptions.

Direct cold tier upserts and partition splits enabling concurrent streaming data ingestion for time series databases

Methods, systems, and computer-readable media for a scalable architecture for a distributed time-series database are disclosed. Using a fleet of ingestion routers, time-series data generated by a plurality of client devices is stored into a plurality of durable partitions. The time-series data comprises a plurality of time series, and an amount of the ingestion routers is determined based at least in part on an ingestion rate of the time-series data. Using a fleet of stream processors, the time-series data from the durable partitions is stored into a plurality of storage tiers including a first storage tier and a second storage tier. A retention period for the first storage tier differs from a retention period for the second storage tier. An amount of the stream processors is determined based at least in part on the time-series data in the durable partitions.

Scalable architecture for a distributed time-series database

Methods, systems, and computer-readable media for versioned metadata management for a time-series database are disclosed. A metadata service of a distributed time-series database stores, to one or more metadata storage resources, first metadata descriptive of a table. The first metadata is associated with a version number in a logical sequence for the table. The metadata service stores, in a log, data indicative of one or more metadata updates for the table. The metadata update(s) are associated with an additional version number higher than the version number in the logical sequence. Responsive to a read request, the metadata service determines a most recent version number that has been applied from the log to the metadata storage resource(s). Responsive to the read request and based (at least in part) on the most recent version number, the metadata service returns either the first metadata or second metadata comprising the update(s).

Versioned metadata management for a time-series database

Methods, systems, and computer-readable media for continuous functions in a time-series database are disclosed. A plurality of data points of a time series are stored into one or more storage tiers of a time-series database. The plurality of data points comprise a plurality of discrete measurements at respective timestamps. Using one or more query processors of the time-series database, a query of the time series is initiated. The query indicates a time range. Using the one or more query processors, a continuous function is determined that represents a segment of the time series in the time range. The continuous function is determined based at least in part on the plurality of data points. An operation is performed using the continuous function as an input.

Continuous functions in a time-series database

A system that implements a data storage service may store data for a database table in multiple replicated partitions on respective storage nodes. In response to a request to back up a table, the service may back up individual partitions of the table to a remote storage system independently and (in some cases) in parallel, and may update (or create) and store metadata about the table and its partitions on storage nodes of the data storage service and/or in the remote storage system. Backing up each partition may include exporting it from the database in which the table is stored, packaging and compressing the exported partition for upload, and uploading the exported, packaged, and compressed partition to the remote storage system. The remote storage system may be a key-value durable storage system in which each backed-up partition is accessible using its partition identifier as the key.

Backup of partitioned database tables

A system that implements a data storage service may store data on behalf of clients in multiple replicas on respective computing nodes. The system may employ an external service to select a master replica for a replica group. The master replica may service consistent read operations and/or write operations that are directed to the replica group (or to a data partition stored by the replica group). The master replica may employ a quorum based mechanism for performing replicated write operations, and a local lease mechanism for determining the replica authorized to perform consistent reads, even when the external service is unavailable. The master replica may propagate local leases to replica group members as replicated writes. If another replica assumes mastership for the replica group, it may not begin servicing consistent read operations that are directed to the replica group until the lease period for a current local lease expires.

System and method for maintaining a master replica for reads and writes in a data store

A platform for data analytics may be provided in a hosted environment on a multi-tenant system. The platform provider may also provide transactional processing services. Data obtained from processing the transactional services may be stored in an n-dimensional cube with which analytics may be performed. A dimension and hierarchy model may be identified based on correlations between hierarchy dimensions and levels in a dataset, or in schema and queries related to the dataset. Correlations may be further based on data received from a data stream. Priority for calculating a hierarchy may be based on data received from a data stream.

Automated hierarchy detection for cloud-based analytics

Prediction logic analyzes previous data usage activities of a customer process running on a host machine to generate a first prediction indicating that the customer process will request a first data set at a first time. The prediction logic retrieves the first data set from long-term storage and loads the first data set into memory on the host machine in advance of the first time in order to provide the customer process with access to first data set in the memory during a period between the first time and a second time. The prediction logic further generates a second prediction indicating that the customer process will not access the first data set for a threshold period of time after the second time and stores the first data set in the long-term storage at the second time.

Predictive loading and unloading of customer data in memory

Distributed database management systems may maintain collections of items spanning multiple partitions. Index structures may correspond to items on one partition or to items on multiple partitions. Item collections and indexes may be replicated. Changes to the data maintained by the distributed database management system may result in updates to multiple index structures. The changes may be compiled into an instruction set applicable to the index structures. In-memory buffers may contain the instructions prior to transmission to affected partitions. Replication logs may be combined with an acknowledgment mechanism for reliable transmission of the instructions to the affected partitions.

Index update pipeline

A system that implements a scalable data storage service may maintain tables in a data store on behalf of storage service clients. The service may maintain table data in multiple replicas of partitions that are stored on respective computing nodes in the system. In response to detecting an anomaly in the system, detecting a change in data volume on a partition or service request traffic directed to a partition, or receiving a service request from a client to split a partition, the data storage service may create additional copies of a partition replica using a physical copy mechanism. The data storage service may issue a split command defined in an API for the data store to divide the original and additional replicas into multiple replica groups, and to configure each replica group to maintain a respective portion of the table data that was stored in the partition before the split.

System and method for performing live partitioning in a data store

Methods and apparatus for conditional master election in a distributed database are described. A plurality of replicas of a database object are stored by a distributed database service. Some types of operations corresponding to client requests directed at the database object are to be coordinated by a master replica. Client access to the database object is enabled prior to election of a master replica. In response to a triggering condition, a particular replica is elected master. The master coordinates implementation of operations with one or more other replicas in response to client requests.

Conditional master election in distributed databases

Growing community of inventors

Seattle, WA, United States of America

Timothy Andrew Rath