Splunk Inc.

Cisco Technology, Inc.

Splunkinc.

Igor Stojanovski

Ledion Bitincka

Alexandros Batsakis

Steve Yu Zhang

Sai Krishna Sajja

Stephen Phillip Sorkin

Eric Timothy Woo

Ashish Mathew

Tameem Anwar

Sourav Pal

Zhenghui Xie

Mehul Goyal

Tianyi Gou

Douglas Rapp

Anish Shrigondekar

Paul J Lucas

Steve Wong

Vishal Patel

Ruochen Zhang

Dhruva Kumar Bhagi

Sophia Rui Zhu

Bharath Aleti

Kartheek Babu Kolla

Esguerra Ma Kharisma

Kai-Sern Lim

Arindam Bhattacharjee

Christopher Madden Pride

Alexander Douglas James

David Marquardt

David Everett Simmen

Thomas Allan Haggie

Clint Sharp

Bharath Kishore Reddy Aleti

Hasan Alayli

James Monschke

Hailun Yan

Murugan Kandaswamy

Marc Andre Chene

Srinivas Bobba

Amrittpal Singh Bath

Pratiksha Shah

Alexandras Batsakis

John Nguyen

Sai Krishna

Srinivas Chowdhary Bobba

A data intake and query system can manage the search of large amounts of data using one or more processing nodes. When a new processing node is added or becomes available, the node coordinator can reassign duties from one or more processing nodes to the new processing node. The node coordinator can initially assign the new processing node one or more groups of data for backup purposes. At a later time, the node coordinator can reassign the new processing node to the one or more groups of data for searching purposes.

Reassignment of processing nodes

Systems and methods are disclosed for processing and executing queries in a data intake and query system. The data intake and query system receives raw machine data at an indexing system, and stores at least a portion of the raw machine data in buckets using containerized indexing nodes instantiated in a containerized environment. The data intake and query system stores the buckets in a shared storage system.

Processing data using nodes in a scalable environment

A data intake and query system can manage the search of large amounts of data using one or more processing nodes. The data intake and query system can identify a first group of processing nodes and cause a first processing node of the group to download and search a particular data group based on a first node map. The data intake and query system may identify a second group of processing nodes that includes the first group of processing nodes and a second processing node. The data intake and query system can transmit commands to cancel one or more data group downloads at the first processing node and can reassign the corresponding data groups for download to the second processing node based on a second node map.

Reassigning data groups based on activation of a processing node

A data intake and query system can ingest and index large amounts of data using one or more ingestors and indexers. The ingestors can ingest incoming data, use it to generate events, and communicate the generated events to a message bus. Indexers can monitor their capacity to process additional groups of events. Based on a determination that an indexer has capacity to process one or more groups of events, the indexer can request one or more messages from the message bus, and process the events associated with the one or more messages.

Generating and storing aggregate data slices in a remote shared storage system

A data intake and query system can manage the search of large amounts of data using one or more processing nodes. The data intake and query system can identify a group of processing nodes and assign a first processing node of the group to download and search a particular data group based on a first node map. The data intake and query system may identify an action associated with the first processing node. The data intake and query system can cause a particular processing node of the group to download the particular data group based on a second node map and transmit an authorization to perform the action to the first processing node.

Deactivating a processing node based on assignment of a data group assigned to the processing node

Reassigning a processing node from downloading to searching a data group

Techniques and mechanisms are disclosed to optimize the size of index files to improve use of storage space available to indexers and other components of a data intake and query system. Index files of a data intake and query system may include, among other data, a keyword portion containing mappings between keywords and location references to event data containing the keywords. Optimizing an amount of storage space used by index files may include removing, modifying and/or recreating various components of index files in response to detecting one or more storage conditions related to the event data indexed by the index files. The optimization of index files generally may attempt to manage a tradeoff between an efficiency with which search requests can be processed using the index files and an amount of storage space occupied by the index files.

Reducing index file size based on event attributes

Systems and methods are described for monitoring indexing nodes, populating and maintaining a resource catalog with relevant information, receiving requests for indexing node availability or assignments, identifying indexing nodes that are available to process data, and/or communicating information relating to available indexing nodes. The system can maintain the resource catalog based on communications with each of the containerized indexing nodes. The system can receive, from a partition manager of a data intake and query system, a request for a containerized indexing node that the partition manager can assign to process data received by the partition manager. The system can identify an available containerized indexing node to process the data. The system can communicate, to the partition manager, an indexing node identifier associated with the available containerized indexing node.

Identifying an indexing node to process data using a resource catalog

Systems and methods are described for improving data availability and/or resiliency of indexers of a data intake and query system. Due to a lag between the time at which data is received and the time at which the data is available for searching, the data intake and query system may receive a query indicating that received (but unavailable for search) data is to be included as part of the query. A cluster master can dynamically track what data is available for searching by different indexers and map the data to filter criteria using a bucket map identifier. When a search head receives a query, it can request a bucket map identifier from the cluster master and send the bucket map identifier to the indexers that will be executing the query. The indexers can use the bucket map identifier to request the individual buckets that they are assigned to search.

Mapping buckets and search peers to a bucket map identifier for searching

Systems and methods are disclosed for making space available in a local storage of a data intake and query system. A cache manager of the data intake and query system may determine an amount of storage space of a local data store that is available for use to perform a query. The cache manager may then use one or more eviction policies associated with content stored at the local data store to purge content items to evict from the local storage. The system may then retrieve content for performing the query from a remote storage and store the retrieved content at the local storage.

Evicting data associated with a data intake and query system from a local storage

Systems and methods are disclosed for processing and executing queries in a data intake and query system. The data intake and query system receives raw machine data at an indexing system, and stores at least a portion of the raw machine data in buckets. Based on a determination that the size of multiple buckets satisfies a threshold size, the data intake and query system converts the buckets to non-editable buckets and stores the data in a remote shared storage system.

Managing and storing buckets to a remote shared storage system based on a collective bucket size

Reassigning data groups from backup to searching for a processing node

Systems and methods are disclosed for processing and executing queries in a data intake and query system. An indexing system of the data intake and query system receives data and stores at least a portion of it in buckets, which are then stored in a shared storage system. The indexing system merges multiple buckets to generate merged buckets and uploads the merged buckets to the shared storage system.

Merging buckets in a data intake and query system

Systems and methods are described for improving data availability and/or resiliency of indexers of a data intake and query system. A data intake and query system can index large amounts of data using one or more indexers. An indexer can store a copy of the data that the indexer is assigned to process in the shared storage system, and a cluster master can track the storage of the data and the indexer assigned to process the data. In the event an indexer fails or is otherwise unable to index data that it has been assigned to index, the cluster master can assign one or more second indexers to process the data. The second indexer can download the data from the shared storage system.

Recovering pre-indexed data from a shared storage system following a failed indexer

Growing community of inventors

San Francisco, CA, United States of America

Igor Stojanovski