Amazon Technologies, Inc.

Gary Taylor

Brian Dennehy

Sebastien Alexandre Roland Rodriguez

Julio Delgado Mangas

Andrea Giuliano

Gavin Bramhill

Richard Nel-Boland

Shane Stacey

Carlo Bergonzini

Hui Zhang

Michele Ficarra

Described technologies generate a data structure corresponding to values sequenced based on a plurality of timestamps associated with the values. The data structure can include a first section identifying a first timestamp associated with the plurality of timestamps and a number representing how many timestamps are associated with the plurality of timestamps, and a second section including at least a value linked to the first timestamp, and an additional value representing an encoding type associated with the second section. The data structure can be stored in computer-implemented storage.

Metric data processing and storage

Data being identified includes a first portion of data and a second portion of data. Based on identifying the data, a data structure is generated. The data structure can include a first section having a first symbol associated with the first portion of data and a second symbol associated with the second portion of data. Further, the first section can include a first offset value corresponding to the first portion of data and a second offset value corresponding to the second portion of data. The data structure can include a second section with a plurality of pointers that reference at least a plurality of symbols including at least the first and second symbol. The data structure can be referenced to process one or more queries against the data.

A query including a time range can be received. The query can be processed to determine data associated with the time range of the query. Cached data can be analyzed to identify a first block of data including metadata identifying a first timestamp associated with the time range of the query. A communication can be generated to include the first timestamp associated with the first block of data. Based on the communication, a second block of data can be received with metadata identifying a second timestamp associated with the time range of the query. A query result can be generated to include the first block of data and the second block of data.

By obtaining metadata for transactions submitted by a service to a log-first distributed database of a provider network, a metrics manager may determine database replica performance for those transactions and notify clients of potential performance issues. When an instance of the service submits a write transaction to the log-first distributed database, the transaction may include a host name and a timestamp for the submission of the transaction. At a later point in time, a write applier may obtain the transaction and apply it to a local database replica, along with an additional timestamp for the application of the transaction to the replica. A metrics manager may obtain the transaction timestamps from the replica and calculate a latency metric for the propagation of the transaction from the particular service instance/instance host to the replica. The latency metric may be stored or transmitted to an endpoint (e.g., a client or administrator).

Replica performance for transactions submitted to log-first distributed databases

Systems and methods for generating dynamic sparse exponential histograms. The system includes a network-based service and a data compression engine to generate a sparse exponential histogram (SEH) representation of a distribution of a plurality of data values of a performance metric of the network-based service. The data compression engine is configured to, map each data value to a bin of an exponential histogram. Responsive to a determination that the mapped bin is not indicated in the SEH representation and that a bin quantity limit would be exceeded by adding the mapped bin, the data compression engine is configured to increase a bin size parameter by a scaling factor to expand data value ranges of the bins, merge bins indicated in the SEH representation according to the expanded data value ranges to reduce the quantity of bins indicated in the SEH representation, and indicate the scaling factor in the SEH representation.

System and method for generating dynamic sparse exponential histograms

A computing resource service receives a request to retrieve data, wherein fulfillment of the request involves retrieval of representations of the data from a number of datastores maintained by the service. The service transmits requests to individual datastores to retrieve the representations of the data. Before all responses to these requests are received, the service evaluates any received responses to determine whether a quorum has been reached with regard to the representations of the data. If so, the service consolidates the representations of the data and provides the consolidated data to fulfill the request.

Data storage and retrieval in redundant storage systems

A first value of a first data type is obtained as input. A second value of a second data type is obtained based at least in part on the first value. The second value is determined to match the first value without losing precision. The second value is determined to be greater than a maximum value for a third data type, and less than or equal to a maximum value for a fourth data type. A marker value is stored in a first storage location, the marker value indicating that a second storage location holds a value of the fourth data type. The second value is stored in the second storage location as the fourth data type.

System for optimizing serialization of values

A computing resource monitoring service receives a request to store a measurement for a metric associated with a computing resource. The request includes the measurement itself and metadata for the measurement, which specifies attributes of the measurement. Based at least in part on the metadata, the computing resource monitoring service generates a fully-qualified metric identifier and, using the identifier, selects a logical partition for placement of the measurement. From the logical partition, the computing resource monitoring service transmits the measurement to an aggregator sub-system comprising one or more in-memory datastores. The computing resource monitoring service stores the measurement in an in-memory datastore within the aggregator sub-system.

Datastore for aggregated measurements for metrics

A computing resource monitoring service receives a request to retrieve measurement data for a metric. The computing resource monitoring service determines, based at least in part on information included in the request, one or more in-memory datastores where the measurement data is stored. If the information indicates that the computing resource monitoring service is to provide an authoritative subset of the measurement data, the computing resource monitoring service limits the data provided in response to the request to the authoritative subset of the measurement data. Otherwise, if the information does not specify this indication, the computing resource monitoring service will provide the measurement data that is available.

Retrieval of authoritative measurement data from in-memory datastores

A computing resource monitoring service receives a request to store metrics data. The request includes the metrics data and a fully qualified metric identifier generated based at least in part on metadata included in an earlier request to store other metrics data for the same metric. The service uses the fully qualified metric identifier to identify an aggregator datastore for storing the metrics data. Once the service has identified the aggregator datastore, the service accesses the aggregator datastore to store the metrics data such that the metrics data and the other metrics data can be obtained in response to a request to obtain data for the metric, where the request includes the metadata.

Aggregation of metrics data with fine granularity

A web server computer system receives a plurality of measurements for a metric from one or more computing resources associated with the web server computer system. Each measurement includes metadata for the measurement, which specifies attributes of the measurement. The web server computer system determines, for each measurement and based at least in part on the metadata, a fully qualified metric identifier for the measurement. The web server computer system uses the fully qualified metric identifier to partition the plurality of measurements into various partitions. Once completed, the web server computer system transmits a request to one or more aggregator sub-systems of a computing resource monitoring service to store the plurality of measurements.

Architecture for metrics aggregation without service partitioning

A computing resource monitoring service receives a plurality of measurements for a metric associated with an auto-scale group. Each measurement is associated with metadata for the measurement, which specifies attributes for the measurement. The computing resource monitoring service determines, for each measurement and based at least in part on the metadata, a fully qualified metric identifier for the measurement. The service partitions the plurality of measurements into a plurality of logical partitions associated with one or more in-memory datastores. The service transmits the measurements from the plurality of logical partitions to the one or more datastores for storage of the measurements. These measurements are provided to one or more computing resource managers for the auto-scale group to enable automatic scaling of computing resources of the group based at least in part on the measurements.

Automatic scaling of computing resources using aggregated metrics

A computing resource monitoring service receives a request to store a measurement for a metric associated with a computing resource. The request includes the measurement itself and metadata for the measurement, which specifies attributes of the measurement. Based at least in part on the metadata, the computing resource monitoring service generates a fully qualified metric identifier and, using the identifier, selects a logical partition for placement of the measurement. From the logical partition, the computing resource monitoring service transmits the measurement to an aggregator sub-system comprising one or more in-memory datastores. The computing resource monitoring service stores the measurement in an in-memory datastore within the aggregator sub-system.

Systems and methods for efficient data aggregation with sparse exponential histograms may support a performance monitor monitoring and recording of one or more performance metrics for one of one or more dimensions. In response to receiving performance data, a performance data aggregator may parse the performance data for one of one or more performance metrics and one or more dimensions. A sparse exponential histogram generator may create a sparse exponential histogram with the parsed performance data. The sparse exponential histogram generator may transmit or store the sparse exponential histogram representation for subsequent analysis.

Growing community of inventors

Dublin, Ireland

Gary Taylor