Splunk Inc.

Cisco Technology, Inc.

Splunkinc.

Joseph Ari Ross

Gergely Danyi

Rajesh Raman

Maxime Petazzoni

Matthew William Pound

Justin Pearson Smith

Steven Karis

Steven Flanders

William Deaderick

Abraham Starosta

Eyal Ophir

Tanner Gilligan

Eric Wohlstadter

Chengyu Yang

Charles Smith

Scott Stewart

Calvin Chan

Houwu Bai

Julien Didier Jean Veron Vialard

Ozan Turgut

Nishant Agarwal

Darshan Patel

Kristal Curtis

Paul Ingram

Sichen Zhong

Sahinaz Safari Sanjani

Sonya Chang

Ramakrishnan Hariharan Chandrasekharapuram

Sunil Kittinakere Nagesh Koundinya

Subhabrata Majumdar

Eric Allen Wohlstadter

Techniques used by an observability system for facilitating near real-time analytics using structures representing or summarizing data distributions, such as histograms. An observability system receives a set of datapoints in which each received datapoint includes data for a structure representing a data distribution. For example, each of the datapoints may include data for a histogram including at least two dimensions. A processing pipeline is disclosed for processing the received data and making the results of the processing available to a user in near real-time. The processing includes identifying, from the datapoints received by the observability system, a subset of datapoints that fall within a time window, generating aggregate data based upon data in the identified subset of datapoints, computing one or more statistical measures based upon the aggregate data, and enabling querying of the subset of datapoints, the aggregate data, and the statistical measures computed for the first minute.

Near real-time analytics using structures representing data distributions

Queries may be resolved against large quantities of collected data (such as traces) by dividing collected data into multiple time intervals and incrementally assigning multiple workers the query and the collected data over the multiple time intervals. For each time interval, these workers may conditionally update one or more summary data structures within the worker based on the query and the portion of collected data assigned to the worker. The summary data structures for each time interval may then be incrementally returned and merged with results from earlier time intervals to create a final merged query result.

Efficiently analyzing trace data

A computerized method is disclosed that includes operations of receiving incoming data including event data, extracting entities from the event data based on a graph ontology, generating a graph-based dense representation of each graph entity according to the graph ontology, wherein the graph-dense representations are stored in a vector database, computing relatedness scores between each of the entities, generating a listing of events related to a selected event, wherein the listing of events is ordered by corresponding relatedness scores, generating a graphical user interface illustrating the listing of events related to the selected event, and causing rendering of the graphical user interface on a display screen of a network device. Generating the graph-based dense representations may include training a graph neural network model on a corpus of metapaths to produce node embeddings.

Automated provision of a listing of events related and corresponding attributes related to a selected event through generation of graph-based dense representations of events of a nodal graph

Implementations of this disclosure provide an anomaly detection system that automatically tunes parameters of a forecasting detector that detects anomalies in a metric time series. The anomaly detection system may implement a three-stage process where a first stage tunes a historical window parameter, a second stage tunes a current window parameter, and a third stage tunes the number of standard deviation different from historical mean required to trigger an alert. The tuned historical window length determined by the first stage may be provided to the second stage as input. Both the tuned historical window length and the tuned current window length may be provided to the third stage as input as use in determining the tuned number of standard deviations.

Automated determination of tuned parameters for analyzing observable metrics

A method of generating metrics data associated with a microservices-based application comprises ingesting a plurality of spans and mapping an ingested span of the plurality of spans to a span identity, wherein the span identity comprises a tuple of information identifying a type of span associated with the span identity, wherein the tuple of information comprises user-configured dimensions. The method further comprises grouping the ingested span by the span identity, wherein the ingested span is grouped with other spans from the plurality of spans comprising a same span identity. The method also comprises computing metrics associated with the span identity and using the metrics to generate a stream of metric data associated with the span identity.

Generating span related metric data streams by an analytic engine

Implementations of this disclosure provide an anomaly detection system and methods of performing anomaly detection on a time-series dataset. The anomaly detection may include utilization of a forecasting machine learning algorithm to obtain a prediction of points of the dataset and comparing the predicted value of a point in the dataset with the actual value to determine an error value associated with that point. Additionally, the anomaly detection may include determination of a sensitivity threshold that impacts whether points within the dataset associated with certain error values are flagged as anomalies. The forecasting machine learning algorithm may implement a seasonality component determination process that accounts for seasonality or patterns in the dataset. A search query statement may be automatically generated through importing the sensitivity threshold into a predetermined search query statement that implements that forecasting machine learning algorithm.

Hyperparameter tuning for anomaly detection service implementing machine learning forecasting

Described are systems, methods, and techniques for collecting, analyzing, processing, and storing time series data and for evaluating and dynamically estimating a resolution of one or more streams of data points and updating an output resolution. Responsive to receiving a stream of data points, a data resolution can be derived and an output resolution can be set to a first value. When a change to the data resolution is detected, the output resolution can be changed, modifying a frequency at which output data points are generated and/or transmitted. In some instances, a detector can be implemented to trigger an alert responsive to ingested data points corresponding with triggering parameters. An output resolution for the detector can be dynamically modified based on dynamically detecting a change to the data resolution of the stream of data.

Dynamic resolution estimation for a detector

Dynamic resolution estimation in metric time series data

An instrumentation analysis system processes data streams received from servers executing instrumented software. The system determines a set of servers that satisfy a given criteria, for example, a set of servers with high resource utilization. The set of servers may be determined by the system based on triggers or specified by a user. The system analyzes properties of servers to determine a property that characterizes the set of servers. The property characterizing the servers is provided to users via a user interface or alerts for further analysis, for example, to analyze the cause of high resource utilization.

Analyzing servers based on data streams generated by instrumented software executing on the servers

Described are systems, methods, and techniques for collecting, analyzing, processing, and storing time series data and for evaluating and determining whether and how to include late or delayed data points for inclusion when publishing or storing the time series data. Maximum delay values can identify a duration for waiting for late or delayed data, such as prior to publication. In some examples, maximum delay values can be dynamically adjustable based on a statistical evaluation process. For late or delayed data points that are received after the maximum delay elapses, some data points can be included in the stored time series data, such as if they are received in the same order that they are generated.

Quantization of data streams of instrumented software and handling of delayed data by adjustment of a maximum delay

A system receives a time series of data values from instrumented software executing on an external system. Each data value corresponds to a metric of the external system. The system stores a level value representing a current estimate of the time series and a trend value representing a trend in the time series. The level and trend values are based on data in a window having a trailing value. In response to receiving a most recent value, the system updates the level value and the trend value to add an influence of the most recent value and remove an influence of the trailing value. The system forecasts based on the updated level and trend values, and in response to determining that the forecast indicates the potential resource shortage event, takes action.

Processing data streams received from instrumented software using incremental finite window double exponential smoothing

A method of normalizing URLs associated with a real user session comprises extracting uniform resource locators (URLs) from ingested spans where at least a portion of the URLs comprise unique URL strings. The method also comprises decomposing each of the URLs into a sequence of tokens and grouping together subsets of related URLs. Also, the method comprises representing each subset of related URLs with a normalized URL string.

URL normalization for rendering a service graph

A process for analyzing an incident includes setting up an alert for a high error rate on a particular endpoint. Once the alert is triggered, a set of traces for transactions exhibiting errors on the offending endpoint is queried. All traces for other services/operations that include errors on the offending endpoint are also enumerated. A set of baseline transactions that involve the offending endpoint, but do not result in error may be utilized to determine whether the errors are always present, or are distinctive for certain offending transactions. All traces are ranked based on a statistic. Once the traces have been ranked, they may be traced down to a deepest/most terminal error. A set of transactions that correlate to the terminal error may also be analyzed to determine infrastructure causes.

Identifying a root cause of an error

A computer-implemented method for analyzing spans and traces associated with a microservices-based application executing in a distributed computing environment comprises aggregating a plurality of ingested spans associated with one or more applications executing in the distributed computing environment into a plurality of traces, wherein each of the plurality of ingested spans is associated with a plurality of tags. The method further comprises comparing durations of a set of related traces of the plurality of traces to determine patterns for the plurality of tags and generating a histogram that represents a distribution of the durations of the set of related traces. The method also comprises providing alerts for one or more tags from the plurality of tags associated with traces having a duration above a threshold based on the distribution of the durations.

Configuring alerts for tags associated with high-latency and error spans for instrumented software

Embodiments of the present disclosure provide for trace and span sampling and analysis for instrumented software. Each span may be annotated with one or more tags that provide context about an executed task, such as a user instrumenting the software, a document involved in a request, an infrastructure element used in servicing a request, etc. A sampler may perform tail-based sampling of traces comprising spans. The sampler may select a portion of the traces having selected features and send them to an analyzer. The analyzer may receive the selected traces and determine whether the selected traces are indicative of configuration problems for the instrumented software. An alert may be generated based on identified configuration problems.

Growing community of inventors

Redwood City, CA, United States of America

Joseph Ari Ross