Amazon Technologies, Inc.

Emmanuel Pierre Devillard

Daniel Vassallo

Hongqi Wang

Muhammad Umer Azad

Robert Mark Waugh

Nitin Kesarwani

Greg Sterin

Wei Huang

Sheikh Naveed Zafar

Henry David Hahn

Xiaosong Pan

Systems and methods are described for the collection and transmission of virtual machine resource operations information. Individual agents on virtual machine resources collect and store operations information in accordance with a current operations information collection configuration. The individual agents will initiate a transmission of the collected operations information, including an identification of the current operations information collection configuration being implemented by the respective agent. Responsive to the receipt of the transmission of the collected operations information, the monitoring processing service makes a determination of whether the current operations information collection configuration is appropriate or should otherwise be maintained or changed. The monitoring processing service will include the identification of the new operations information collection configuration in the message responsive to the transmission of the collected operations information from the agent.

Dynamically configurable operation information collection

A leader host obtains individual distributions of data sets ingested by individual hosts of a fleet of hosts over a domain. The leader host compiles the individual distributions over the domain to generate a compiled distribution. The leader host then partitions the domain based at least in part on the generated compiled distribution. These partitions of the partitioned domain are distributed to individual hosts of the fleet of hosts, which causes the individual hosts to process a portion of the distributed date set according to their respective partitions.

Horizontal scaling of time series data

A system for collecting and disseminating log information may comprise a log agent and a log service. A log agent may transmit a log stream to a log service using non-overlapping adjacent subsets of log entries. Each subset may be associated with a unique and monotonically increasing sequence number, which is not incremented until the subset is received and processed by a log service.

Log information transmission integrity

A near real time data tailing mechanism can enable data, received on a stream, to be directed to a specified location independent of any processing and persistent storage of that data. A customer can submit a tail request that can be received to a front end of a data management service. When a request is received to store data to the persistent storage, a determination is made that the front end has registered a tail for that data stream and the host receiving the data can forward a copy of the data to the front end, which can cause the data to be transmitted to the specified location. One or more filters can be applied in order to cause only specific data to be transmitted for the tail request. A best effort approach provides an overview of the data in near real time.

Dynamic data tailing

A technology is provided for enabling a partitioned search to be performed on log events from multiple log streams that are stored by multiple hosts. A search query may be submitted to identify the log streams whose log events are to be searched and to indicate a time interval in which log events are to have occurred as indicated by the log events' time stamps. The multiple hosts may search stored log events in parallel and return a set of log-event search results satisfying the search query. A pagination token can be included with the set of log event search results. The pagination token may be used to resume the search if the multiple hosts were not able to completely finish searching the stored log events before the set of log-event search results had to be returned to prevent a timeout of a search client.

Partitioned search of log events

Systems and methods are provided for organizing data channels and processing hosts included in a system into clusters. A cluster management service may receive data from a steam of data and may route the data to a cluster associated with the data stream. A data channel routing service included in the cluster may route the data to the set of processing hosts included in the cluster through multiple data channels included in the cluster. In some instances, the data channel routing service may use any of the data channels to send data to the set of processing hosts. Because incoming data may be distributed among multiple data channels, the cluster may experience less congestion. Further, the system may also process the stream of data using the same processing hosts by routing the stream of data to the same cluster, thereby avoiding split processing of the data stream.

Clustered architecture design

A log data service in a virtual environment that allows customers of a compute service provider to access system, application and custom log files associated with virtual machine instances that are executing. In some embodiments, log data can be received that includes events or messages from virtual machines in a multi-tenant environment. The log data can be transformed into metric data, which can be used by the customer to generate statistics, view graphs, and generally monitor the customer's virtual machine instances. The log data can also be stored as a service so that the customer has a central repository for which to access the log data.

Growing community of inventors

Issaquah, WA, United States of America

Emmanuel Pierre Devillard