Vmware, Inc.

Decho Corporation

Emc IP Holding Company LLC

University of Washington

Nicholas Kushmerick

Junyuan Lin

Darren Brown

Ashot Nshan Harutyunyan

Arnak Poghosyan

Naira Movses Grigoryan

Matt Roy McLaughlin

Vardan Movsisyan

Mayank Agarwal

Jon Herlocker

Dahlia Malkhi

Steven Flanders

Paul Pedersen

Eric Schkufza

Keshav Mathur

Peng Gao

Xing Wang

Ramses V Morales

Jinyi Lu

Harutyun Beybutyan

Daniel S Weld

Jeremy OlmstedThompson

Robert B Doorenbos

Patrick Edward Bozeman

Udi Wieder

Dhaval Gada

Computational methods and systems for detecting and troubleshooting anomalous behavior in distributed applications executing in a distributed computing system are described herein. Methods and systems discover nodes comprising the application. Anomaly detection monitors the metrics associated with the nodes for anomalous behavior in order to identify an approximate point in time when anomalous behavior begins to adversely impact performance of the application. Anomaly detection also monitors logs messages associated with the nodes to detect anomalous behavior recorded in the log messages. When anomalous behavior is detected in either the metrics and/or the log messages an alert identifying the anomalous behavior is generated. Troubleshooting guides an administrator and/or application owner to investigate the root cause of the anomalous behavior. Appropriate remedial measures may be determined based on the root cause and automatically or manually executed to correct the problem.

Methods and systems for troubleshooting applications using streaming anomaly detection

Automated processes and systems for detecting abnormally behaving objects of a distributed computing system are described. Processes and systems obtain metrics that are generated in a historical time window and are associated with an object of the distributed computing system. Processes and system use the metrics to compute a time-dependent system indicator over the historical time window. Each value of the system indicator corresponds to a point in time of the historical time window when the object was in a normal or an abnormal state. Processes and systems use the normal and abnormal states of the system indicator in the historical time window to train a state classifier that is used to detect run-time abnormal behavior of the object. When the state classifier identifies abnormal behavior of the object, an alert is generated, indicating the abnormal behavior of the object.

Processes and systems that detect abnormal behavior of objects of a distributed computing system

Methods and systems automatically adjusting resources and monitoring configurations of objects of a distributed computing system in response to changes to application programs. Methods search event messages for information indicating a change in execution of an object. The information is used to determine resource allocation rules of infrastructure resources by and a monitoring configuration for the object. Expected impacts on the infrastructure resource are determined from the rules. When an expected impact is greater than an associated impact threshold, use of the infrastructure resources may be adjusted to accommodate the changes. The adjustments include scaling up or down the infrastructure resources. When the object is a virtual object, the virtual object may be migrated from one server computer to another server computer within the distributed computer system. The monitoring configuration is used to adjust tools that monitor the objects of the distributed computing system.

Methods and systems to adjust resources and monitoring configuration of objects in a distributed computing system

Methods and systems to sample event messages are described. As event messages are generated by one or more sources, the event messages are stored in a storage queue. An event message policy that represents conditions for storing event messages in a sample log file are input. For each event message output from the storage queue, the event message may be stored in a sample log file when one or more of the conditions of the event message policy are satisfied. The event messages of the sample log file may be displayed in a graphical user interface that enables a user to change the event message policy.

Methods and systems to sample event messages

The current document is directed to methods and systems that process, classify, efficiently store, and display large volumes of event messages generated in modern computing systems. In a disclosed implementation, received event messages are assigned to event-message clusters based on non-parameter tokens identified within the event messages. A parsing function is generated for each cluster that is used to extract data from incoming event messages and to prepare event records from event messages that more efficiently and accessible store event information. The parsing functions also provide an alternative basis for assignment of event messages to clusters. Event types associated with the clusters are used for gathering information from various information sources with which to automatically annotate event messages displayed to system administrators, maintenance personnel, and other users of event messages.

Method and system for clustering event messages and manage event-message clusters

Automated methods and systems to determine a baseline event-type distribution of an event source and use the baseline event type distribution to detect changes in the behavior of the event source are described. In one implementation, blocks of event messages generated by the event source are collected and an event-type distribution is computed for each of block of event messages. Candidate baseline event-type distributions are determined from the event-type distributions. The candidate baseline event-type distribution has the largest entropy of the event-type distributions. A normal discrepancy radius of the event-type distributions is computed from the baseline event-type distribution and the event-type distributions. A block of run-time event messages generated by the event source is collected. A run-time event-type distribution is computed from the block of run-time event messages. When the run-time event-type distribution is outside the normal discrepancy radius, an alert is generated indicating abnormal behavior of the event source.

Methods and systems to determine baseline event-type distributions of event sources and detect changes in behavior of event sources

Computational processes and systems are directed to detecting abnormally behaving objects of a distributed computing system. An object can be a physical or a virtual object, such as a server computer, application, VM, virtual network device, or container. Processes and systems identify a set of metrics associated with an object and compute an indicator metric from the set of metrics. The indicator metric is used to label time stamps that correspond to outlier metric values of the set of metrics. The metrics and outlier time stamps are used to compute rules by machine learning. Each rule corresponds to a subset or combination of metrics and represents specific threshold conditions for metric values. The rules are applied to run-time metric data of the metrics to detect run-time abnormal behavior of the object.

Processes and systems that detect object abnormalities in a distributed computing system

The current document is directed to systems, and methods incorporated within the systems, that carry out probability-distribution-based analysis of log-file entries. A monitoring subsystem within a distributed computer system uses probability-distribution-based analysis of log-file entries to detect changes in the state of the distributed computer system. A log-file-analysis subsystem within a distributed computer system uses probability-distribution-based analysis of log-file entries to identify subsets of log-file entries that predict anomalies and impending problems in the distributed computer system. In many implementations, a numerical comparison of probability distributions of log-file-entry types is used to detect state changes in the distributed computer system.

Probability-distribution-based log-file analysis

The current document is directed to methods and systems for detecting the occurrences of abnormal events and operational behaviors within the distributed computer system. The currently described methods and systems continuously collect metric data from various metric-data sources, generate a sequence of metric-data observations, each metric-data observation comprising a set of temporally aligned metric data, and employ principle-component analysis to transform the metric-data observations to facilitate reduction of the dimensionality of the metric-data observations. The currently described methods and systems then employ clustering methods to identify outlying transformed-metric-data observations, accordingly label the transformed metric-data observations to generate a training dataset, and then apply one or more of various types of machine-learning techniques to the training dataset in order to generate an abnormal-observation detector that can be used to detect, in real time, abnormal metric-data observations as they are generated within the distributed computing system.

Methods and systems that detect and classify incidents and anomalous behavior using metric-data observations

The current document is directed to automated methods and systems that employ unsupervised-machine-learning approaches as well as rule-based systems to discover distributed applications within distributed-computing environments. These automated methods and systems provide a basis for higher-level distributed-application administration and management tools and subsystems that provide distributed-application-level user interfaces and operations. In one implementation, the currently disclosed methods and systems employ agents within virtual machines that execute routines and programs and that together comprise a distributed application to continuously furnish information about the virtual machines to a pipeline of stream processors that collect and filter the information to provide for periodic application-discovery. The stream processors generate data representations of the processes currently running on the virtual machines and data representations of the communications connections between the virtual machines. An application-discovery subsystem periodically employs these data representations, and additional data derived from them, to identify the different distributed applications running within a distributed-computing facility and to identify tiers of virtual-machine nodes within each identified distributed application. This, in turn, allows the application-discovery subsystem to generate sets of delta changes for the discovered applications after each periodic application discovery.

Methods and systems for identifying application components in distributed computing facilities

The current document is directed to methods and systems that process, classify, efficiently store, and display large volumes of event messages generated in modern computing systems. In a disclosed implementation, event messages are assigned types and transformed into event records with well-defined fields that contain field values. Recurring patterns of event messages, referred to as 'transactions,' are identified within streams or sequences of time-associated event messages and streams or sequences of time-associated event records.

Method and system for identifying event-message transactions

The current document is directed to systems, and methods incorporated within the systems, that execute queries against log-file entries. A monitoring subsystem within a distributed computer system uses query results during analysis of log-file entries in order to detect changes in the state of the distributed computer system, identify problems or potential problems, and predict and forecast system characteristics. Because of the large numbers of log-file-entry containers that may need to be opened and processed in order to execute a single query, and because opening and reading through the entries in a log-file-entry container is a computationally expensive and time-consuming operation, the currently disclosed systems employ event-type metadata associated with log-file-entry containers to avoid opening and reading through the log-file entries of log-file-entry containers that do not contain log-file entries with event types relevant to the query.

Efficient log-file-based query processing

This disclosure is directed to tagging tokens or sequences of tokens in log messages generated by a logging source. Event types of log messages in a block of log messages are collected. A series of tagging operations are applied to each log message in the block. For each tagging operation, event types that are qualified to receive the corresponding tag are identified. When a log message is received, the event type is determined and compared with the event types of the block in order to identify a matching event type. The series of tagging operations are applied to the log message to generate a tagged log message with the restriction that each tagging operation only applies a tag to token or sequences of tokens when the event type is qualified to receive the tag. The tagged log message is stored in a data-storage device.

Methods and systems to tag tokens in log messages

Methods and systems adjust resources and monitoring configuration of a monitoring tool and auxiliary servers of a distributed computing system. Any new virtual object installed in the distributed computing system or change in functionality of an existing virtual object of the distributed computing system is identified. Expected configuration rules and resource allocation rules are determined for the virtual object. The expected configured rules are used to adjust a monitoring configuration of the monitoring tool. The resource allocation rules are used to adjust the infrastructure resources available to the monitoring tool. When the virtual object creates a change in demand for auxiliary services, corresponding auxiliary servers are installed, removed, or changed in the distributed computing system. Resource allocated rules and expected configuration rules are determined for the auxiliary servers and used to adjust the infrastructure resources and monitoring configuration of the monitoring tool. Methods optimize distributions of virtual objects.

Growing community of inventors

Seattle, WA, United States of America

Nicholas Kushmerick