Cribl, Inc.

Splunk Inc.

Cisco Technology, Inc.

Dritan Bitincka

Ledion Bitincka

Clint Sharp

Ting Wang

Nicholas Robert Romito

Oliver Draese

Micah James Delfino

Qianjie Zhong

Feng Shao

Geng Qin

D Randall Young

Jef Bekes

Cary Glen Noel

Min Zhang

Ken Chen

Gang Tao

Junqing Hao

Elias Haddad

Lai Qiang Ding

Janusz Sobolewski

Paul Avery Dugas

Cary Noel

In some aspects, search functionality is provided in an observability pipeline system. In some implementations, a method of searching remotely-stored data includes receiving a search query at a computer node residing at a node geolocation, the search query representing a request to search data stored at a storage geolocation; identifying compute geolocations each including computing resources, each of the compute geolocations being distinct from the node geolocation and the storage geolocation; obtaining latency data including latency values for the compute geolocations; selecting one of the compute geolocations based on the latency data; by operation of a coordinator agent on the computer node, initiating a dynamic computing resource at the selected compute geolocation, and receiving search results identified by the dynamic computing resource.

Searching remote data in an observability pipeline system

In some aspects, search functionality is provided in an observability pipeline system. In some implementations, a search query is received at a computer node from a leader role of an observability pipeline system. The search query represents a request to search event data at the computer node and includes a first search operator that specifies a system state context criterion and a second search operator that specifies an event criterion. An observability pipeline process is configured according to the search query. Search results are generated based on applying the observability pipeline process at the computer node. A current system state of the computer node that matches the first search operator is determined; and a subset of event data on the computer node that matches the second search operator is identified. The search results including the subset of event data are sent to the leader role.

Targeting system state context in a search process in an observability pipeline system

In some aspects, search functionality is provided in an observability pipeline system. In some implementations, a search method includes receiving a search query from a leader role in an observability pipeline system. The search query represents a request to search event data at a computer resource. An observability pipeline process is configured to perform a search according to the search query, and search results are obtained based on applying the observability pipeline process to the event data. The search results include events from the event data. Provenance information is obtained for each of the events. The provenance information for each event includes an identification of the computer resource and a link to the computer resource. Augmented search results are generated by associating the provenance information with the respective events, and the augmented search results are communicated to the leader role.

Enriching search results with provenance information in an observability pipeline system

In some aspects, search functionality is provided in an observability analysis system. In some implementations, search results are obtained from prior search queries, the search results include a subset of event data generated in a computing environment. Metadata are generated by analyzing the event data to identify a plurality of fields and values for each of the plurality of fields. A partial search query is received through a user interface at a first computer node; the partial search query represents at least a portion of a request to search the event data. Autocomplete suggestions are generated at the first computer node based on the partial search query and the metadata. Each of the autocomplete suggestions includes a suggested field selected from the plurality of fields, a suggested value selected from the values, or both. At least one of the autocomplete suggestions is displayed in the user interface.

Generating search query autocomplete suggestions based on metadata derived from event data

Data payloads from an external data storage system are processed in an observability pipeline system. In some aspects, the observability pipeline system defines a leader role and worker roles. The leader role generates a data discovery task based on configuration information for a data collection task. A worker role executes the data discovery task, which includes communicating with an external data storage system to identify a data payload that is stored on the external data storage system and contains event data that meet event filter criteria. The leader role generates data collection tasks based on the data payload. Worker roles execute the data collection tasks. Executing a data collection task includes: communicating with the external data storage system to obtain a subset of filtered event data from the data payload; and streaming the subset of filtered event data to an observability pipeline process.

Processing data payloads from external data storage systems in an observability pipeline system

Techniques and mechanisms are disclosed that enable collection of various types of data from cloud computing services and the generation of various dashboards and visualizations to view information about collections of cloud computing resources. A user can configure collection of data from one or more cloud computing services and view visualizations using an application platform referred to herein as a cloud computing management application. A cloud computing management application further may be configured to generate and cause display of interactive topology map representations of cloud computing resources based on the collected data, where an interactive topology map enables users to view an intuitive visualization of a collection of computing resources, efficiently cause performance of actions with respect to various resources displayed in the topology map, and analyze the collection of resources in ways that are not possible using conventional cloud computing service management consoles.

Displaying interactive topology maps of cloud computing resources

Load balancing processes are performed in an observability pipeline system comprising a plurality of computing resources. In some aspects, the observability pipeline system defines a leader role and worker roles. A plurality of computing jobs each include computing tasks associated with event data. The leader role dispatches the computing tasks to the worker roles according to a least in-flight task dispatch criteria, which includes iteratively: identifying an available worker role; identifying one or more incomplete computing jobs; selecting, from the one or more incomplete computing jobs, a computing job that has the least number of in-flight computing tasks currently being executed in the observability pipeline system; identifying a next computing task from the selected computing job; and dispatching the next computing task to the available worker role. The worker roles execute the computing tasks by applying an observability pipeline process to the event data associated with the respective computing task.

Load balancing computing resources in an observability pipeline system

In a general aspect, an observability pipeline system includes a pack data processing engine. In some aspects, an observability pipeline system includes data processing engines that are configured according to system default configuration settings and system local configuration settings. A pack file received from a remote computer system contains routes, pipelines, and pack default configuration settings. A pack data processing engine includes the routes and pipelines from the pack file. Pack local configuration settings, defined for the pack data processing engine, inherit at least one of the system default configuration settings and at least one of the pack default configuration settings. The pack local configuration settings are isolated from the system local configuration settings. When pipeline data is processed in the observability pipeline system on the computer system, the pack data processing engine is applied to the pipeline data.

Pack framework in an observability pipeline system

In some aspects, an edge-based data collection system discovers, collects, processes, and forwards data in an observability pipeline system. In some implementations, an edge agent of the observability pipeline system runs on a computer node. The edge agent identifies processes running on the computer node; identifies files on the computer node that the processes have opened for writing; accesses log discovery parameters of the observability pipeline system; selects a plurality of files from the identified files according to the log discovery parameters; generates a list of discovered log files that includes a path and a name for each of the plurality of files; adds the list of discovered log files to a list of monitored log files to be monitored by the observability pipeline system; and then monitors the plurality of files to generate input for the observability pipeline system.

Edge-based data collection system for an observability pipeline system

Techniques and mechanisms are disclosed that enable a data collection system to adaptively control collection of data from one or more external data sources. At a high level, adaptively controlling collection of data from external data sources may include collecting performance information related to one or more data collection nodes and, in response to analyzing the collected performance information, adapting rates at which the data collection nodes send data collection requests to external data sources. Data collection performance information generally may include, but is not limited to, network traffic data, error messages generated by external data sources and/or data collection nodes, computing device performance information, and any other types of information related to a data collection node's ability to collect data from external data sources.

Growing community of inventors

Edgewater, NJ, United States of America

Dritan Bitincka