Amazon Technologies, Inc.

Mustafa Ozan Ozen

Timothy Andrew Rath

Gaurav Saxena

Sandeep Bhatia

Lonnie J Princehouse

Gaurav Gupta

Dumanshu Goyal

Atilim Cetin

Omer Ahmed Zaki

Karthik Gurumoorthy Subramanya Bharathy

Nabanita Maji

Nirmesh Khandelwal

Eric Coll

Nilesh Shahdadpuri

Sen Yue

Methods, systems, and computer-readable media for a scalable architecture for a distributed time-series database are disclosed. Using a fleet of ingestion routers, time-series data generated by a plurality of client devices is stored into a plurality of durable partitions. The time-series data comprises a plurality of time series, and an amount of the ingestion routers is determined based at least in part on an ingestion rate of the time-series data. Using a fleet of stream processors, the time-series data from the durable partitions is stored into a plurality of storage tiers including a first storage tier and a second storage tier. A retention period for the first storage tier differs from a retention period for the second storage tier. An amount of the stream processors is determined based at least in part on the time-series data in the durable partitions.

Scalable architecture for a distributed time-series database

Methods, systems, and computer-readable media for continuous functions in a time-series database are disclosed. A plurality of data points of a time series are stored into one or more storage tiers of a time-series database. The plurality of data points comprise a plurality of discrete measurements at respective timestamps. Using one or more query processors of the time-series database, a query of the time series is initiated. The query indicates a time range. Using the one or more query processors, a continuous function is determined that represents a segment of the time series in the time range. The continuous function is determined based at least in part on the plurality of data points. An operation is performed using the continuous function as an input.

Continuous functions in a time-series database

Atomic partition scheme updates to partition items may be implemented by a time series database. A time threshold may be assigned to partition scheme update so that the time threshold may be applied across a set of ingestion nodes that may apply the partition scheme update the same. A request to store an item with a timestamp less than the time threshold may be stored in one partition of the time series database, while the item may be stored in a different partition of the time series database if the item has timestamp greater than or equal to the time threshold.

Atomic partition scheme updates to store items in partitions of a time series database

Methods, systems, and computer-readable media for resource usage restrictions in a time-series database are disclosed. Elements of a plurality of time series are stored into one or more storage tiers of a time-series database. The time series are associated with a plurality of clients of the time-series database. Execution of tasks is initiated using one or more resources of one or more hosts. The time-series elements represent inputs to the tasks. The tasks comprise a first task and a second task. A usage of the one or more resources by the first task is determined to violate one or more resource usage restrictions. Based at least in part on the usage, one or more actions are performed to modify the execution of the first task. The one or more actions increase an amount of the one or more resources available to the second task.

Resource usage restrictions in a time-series database

Techniques for -memory ingestion for highly available distributed time-series databases are described. A method of in-memory ingestion may include obtaining, by a host of a time series database, time series data from one or more electronic devices, the time series database including a plurality of portions of the time series database spread across a plurality of hosts, the plurality of portions of the time series database including at least one hot portion and a plurality of cold portions, storing the time series data in a volatile storage location associated with the hot portion on the host, detecting an event to store the time series data associated with the hot portion to a non-volatile storage location on the host, and storing the time series data associated with the hot portion to the non-volatile storage location.

In-memory ingestion for highly available distributed time-series databases

Methods, systems, and computer-readable media for partition splitting in a distributed database are disclosed. A partition of data is split into a first sub-partition and a second sub-partition. A first portion of the data is assigned to the first sub-partition, and a second portion of the data is assigned to the second sub-partition. One or more elements of the first portion of the data from the partition and an additional one or more elements of the first portion of the data from the first sub-partition are stored into a first node. One or more elements of the second portion of the data from the partition and an additional one or more elements of the second portion of the data from the second sub-partition are stored into a second node. The partition is prevented from receiving new data after the partition is split into the first and second sub-partitions.

Partition splitting in a distributed database

Methods, systems, and computer-readable media for ingestion partition auto-scaling in a time-series database are disclosed. A first set of one or more hosts divides elements of time-series data into a plurality of partitions. A second set of one or more hosts stores the elements of time-series data from the plurality of partitions into one or more storage tiers of a time-series database. An analyzer receives first data indicative of the resource usage of the time-series data at the first set of one or more hosts. The analyzer receives second data indicative of the resource usage of the time-series data at the second set of one or more hosts. Based at least in part on analysis of the first data and the second data, the analyzer initiates a split of an individual one of the partitions into two or more partitions.

Ingestion partition auto-scaling in a time-series database

Techniques are described for clustering data at the point of ingestion for storage using scalable storage resources. To cluster data at the point of ingestion, a data ingestion and query service uses a multilevel hash tree (MLHT)-based index to map a hierarchy of attribute values associated with each data element onto a point of a MLHT (which itself conceptually maps onto a continuous range of values). The total range of the MLHT is divided into one or more data partitions, each of which is mapped to one or more physical storage resources. A mapping algorithm uses the hierarchy of attribute fields to calculate the position of each data element ingested and, consequently, a physical storage resource at which to store the data element.

Dynamic distributed data clustering using multi-level hash trees

Techniques are described for clustering data at the point of ingestion for storage using scalable storage resources. The clustering techniques described herein are used to cluster time series data in a manner such that data that is likely to be queried together is localized to a same partition, or to a minimal set of partitions if the data set is large, where the partitions are mapped to physical storage resources where the data is to be stored for subsequent processing. Among other benefits, the clustered storage of the data at the physical storage resources can reduce an amount of data that needs to be filtered by many types of queries, thereby improving the performance of any applications or processes that rely on querying the data.

Dynamic distributed data clustering

A data storage and retrieval system receives data points for a time series. The data storage and retrieval system stores the data points for a first portion of the time series using a first data format. Based at least in part on an analysis of queries performed on the first portion of the time series, the data storage and retrieval system determines to store a second portion of the time series using a second data format. The data storage and retrieval system stores subsequently received data points of the time series in the second data format.

Growing community of inventors

Vancouver, Canada

Mustafa Ozan Ozen