Splunk Inc.

Cisco Technology, Inc.

Eric Timothy Woo

Alexandros Batsakis

Igor Stojanovski

Micah James Delfino

Johnvey Hwang

Leonid Budchenko

Stephen Phillip Sorkin

Mitchell Blank, Jr

Tameem Anwar

Vishal Patel

David Carasso

Sai Krishna Sajja

Ledion Bitincka

Geoffrey R Hendrey

Ashish Mathew

Paul J Lucas

Mehul Goyal

Douglas Rapp

Sourav Pal

Steve Wong

Anish Shrigondekar

Ruochen Zhang

Zhenghui Xie

Tianyi Gou

Sophia Rui Zhu

R David Carasso

Sundar Vasan

Hasan Alayli

Liang Han

Mitchell Neuman Blank, Jr

Sai Krishna

Embodiments are directed towards previewing results generated from indexing data raw data before the corresponding index data is added to an index store. Raw data may be received from a preview data source. After an initial set of configuration information may be established, the preview data may be submitted to an index processing pipeline. A previewing application may generate preview results based on the preview index data and the configuration information. The preview results may enable previewing how the data is being processed by the indexing application. If the preview results are not acceptable, the configuration information may be modified. The preview application enables modification of the configuration information until the generated preview results may be acceptable. If the configuration information is acceptable, the preview data may be processed and indexed in one or more index stores.

Processing index data based on generation of index data

Embodiments described herein are directed to facilitating management and storage of configurations. In one embodiment, a request to provide configurations associated with a user-application pair is identified. Based on the user-application pair, a user-defined configuration cache is accessed to obtain user-defined configurations, an application-defined configuration cache is accessed to obtain application-defined configurations, and a system-defined configuration cache is accessed to obtain system-defined configurations. Thereafter, at least a portion of the user-defined configurations, the application-defined configurations, and the system-defined configurations are aggregated or merged to generate a set of configurations associated with the user-application pair. Such a set of configurations associated with the user-application pair are provided for use in performing a task.

Facilitating management and storage of configurations

A data intake and query system can manage the search of large amounts of data using one or more processing nodes. When a new processing node is added or becomes available, the node coordinator can reassign duties from one or more processing nodes to the new processing node. The node coordinator can initially assign the new processing node one or more groups of data for backup purposes. At a later time, the node coordinator can reassign the new processing node to the one or more groups of data for searching purposes.

Reassigning a processing node from downloading to searching a data group

In a computer-implemented method for configuring a distributed computer system comprising a plurality of nodes of a plurality of node classes, configuration files for a plurality of nodes of each of the plurality of node classes are stored in a central repository. The configuration files include information representing a desired system state of the distributed computer system, and the distributed computer system operates to keep an actual system state of the distributed computer system consistent with the desired system state. The plurality of node classes includes forwarder nodes for receiving data from an input source, indexer nodes for indexing the data, and search head nodes for searching the data. Responsive to receiving changes to the configuration files, the changes are propagated to nodes of the plurality of nodes impacted by the changes based on a node class of the nodes impacted by the changes.

Two-way replication of search node configuration files using a mediator node

Embodiments of the present disclosure are directed to an interactive development environment (IDE) interface that provides historical visualization of queries and query result information iteratively and intuitively. According to an embodiment of the present disclosure, a process is provided to generate visualizations of queries and processed query result information in a single, persistent, integrated display. Each query and resultant search data information is presented iteratively in chronological order, and maintain a persistent, viewable history of a search data exploration session.

Interactive development environment for visualization of query result information

Systems and methods are described for monitoring indexing nodes, populating and maintaining a resource catalog with relevant information, receiving requests for indexing node availability or assignments, identifying indexing nodes that are available to process data, and/or communicating information relating to available indexing nodes. The system can maintain the resource catalog based on communications with each of the containerized indexing nodes. The system can receive, from a partition manager of a data intake and query system, a request for a containerized indexing node that the partition manager can assign to process data received by the partition manager. The system can identify an available containerized indexing node to process the data. The system can communicate, to the partition manager, an indexing node identifier associated with the available containerized indexing node.

Identifying an indexing node to process data using a resource catalog

Facilitating event creation via previews

Systems and methods are described for improving data availability and/or resiliency of indexers of a data intake and query system. Due to a lag between the time at which data is received and the time at which the data is available for searching, the data intake and query system may receive a query indicating that received (but unavailable for search) data is to be included as part of the query. A cluster master can dynamically track what data is available for searching by different indexers and map the data to filter criteria using a bucket map identifier. When a search head receives a query, it can request a bucket map identifier from the cluster master and send the bucket map identifier to the indexers that will be executing the query. The indexers can use the bucket map identifier to request the individual buckets that they are assigned to search.

Mapping buckets and search peers to a bucket map identifier for searching

Techniques for scheduling search queries in a computing environment are disclosed. A search query scheduling system associates a first set of queries with a first skew tolerance, the first set of queries scheduled to be performed during a first period, where the first skew tolerance is based on a duration of the first period. The search query scheduling system reschedules a first subset of search queries included in the first set of queries by skewing the first subset of search queries over a first portion of the first period based on the first skew tolerance.

Skewing of scheduled search queries

Replication of search-related configuration customizations across multiple individual configuration files of search heads of a cluster for a consistent user experience. A search head leader of the cluster can receive a first journal entry relating to a first customization of a knowledge object from a first search head of the cluster. The search head leader may determine that the first journal entry references a parent commit journal entry matching a latest commit journal entry in a journal maintained by the search head leader. The first journal entry can be stored in the journal and sent to one or more search heads of the cluster.

Management of journal entries associated with customizations of knowledge objects in a search head cluster

Reassigning data groups from backup to searching for a processing node

Systems and methods are disclosed for processing and executing queries in a data intake and query system. An indexing system of the data intake and query system receives data and stores at least a portion of it in buckets, which are then stored in a shared storage system. The indexing system merges multiple buckets to generate merged buckets and uploads the merged buckets to the shared storage system.

Merging buckets in a data intake and query system

Systems and methods are described for improving data availability and/or resiliency of indexers of a data intake and query system. A data intake and query system can index large amounts of data using one or more indexers. An indexer can store a copy of the data that the indexer is assigned to process in the shared storage system, and a cluster master can track the storage of the data and the indexer assigned to process the data. In the event an indexer fails or is otherwise unable to index data that it has been assigned to index, the cluster master can assign one or more second indexers to process the data. The second indexer can download the data from the shared storage system.

Recovering pre-indexed data from a shared storage system following a failed indexer

Systems and methods are described for processing incoming data. The system can receive, from a first partition manager of a data intake and query system, first data that is associated with a first identifier, and can receive, from a second partition manager of the data intake and query system, second data that is associated with a second identifier. The system can process the first data and store first results of said processing the first data in one or more first buckets associated with the first tenant identifier. The system can process the second data and store second results of said processing the second data in one or more second buckets associated with the second tenant identifier.

Processing data associated with different tenant identifiers

Central repository of configuration files and two-way replication of search node configuration files

Growing community of inventors

San Francisco, CA, United States of America

Eric Timothy Woo