Splunk Inc.

Cisco Technology, Inc.

Sai Krishna Sajja

Alexandros Batsakis

Igor Stojanovski

Tameem Anwar

Eric Timothy Woo

Tianyi Gou

Sourav Pal

Ashish Mathew

Anish Shrigondekar

Ruochen Zhang

Zhenghui Xie

Mehul Goyal

Sophia Rui Zhu

Douglas Rapp

Steve Wong

Bharath Kishore Reddy Aleti

Jiahan Wang

Abhinav Prasad Nekkanti

John Nguyen

Arindam Bhattacharjee

Christopher Madden Pride

Vishal Patel

Ledion Bitincka

Alexander Douglas James

Eric Sammer

Joseph Gabriel Echeverria

Hasan Alayli

James Monschke

Kartheek Babu Kolla

Srinivas Bobba

Kai-Sern Lim

Sanket Murlidhar Varkhede

Paul J Lucas

A data intake and query system can manage the search of large amounts of data using one or more processing nodes. When a new processing node is added or becomes available, the node coordinator can reassign duties from one or more processing nodes to the new processing node. The node coordinator can initially assign the new processing node one or more groups of data for backup purposes. At a later time, the node coordinator can reassign the new processing node to the one or more groups of data for searching purposes.

Reassignment of processing nodes

Systems and methods are disclosed for processing and executing queries in a data intake and query system. The data intake and query system receives raw machine data at an indexing system, and stores at least a portion of the raw machine data in buckets using containerized indexing nodes instantiated in a containerized environment. The data intake and query system stores the buckets in a shared storage system.

Processing data using nodes in a scalable environment

A data intake and query system can ingest and index large amounts of data using one or more ingestors and indexers. The ingestors can ingest incoming data, use it to generate events, and communicate the generated events to a message bus. Indexers can monitor their capacity to process additional groups of events. Based on a determination that an indexer has capacity to process one or more groups of events, the indexer can request one or more messages from the message bus, and process the events associated with the one or more messages.

Generating and storing aggregate data slices in a remote shared storage system

A computer-implemented method for performing cross regional disaster recovery includes receiving event data at an active deployment of a data intake and query system. The method further includes processing the event data in a plurality of stages of the data intake and query system to convert the event data into searchable buckets of indexed data, wherein files generated at each stage of the processing are uploaded to a first scalable storage module in the active deployment. Further, the method includes replicating the files generated at each stage of the processing to a second scalable storage module in a standby deployment of the data intake and query system. Responsive to an outage at the active deployment, the method includes transitioning control to the standby deployment and recovering the event data at the standby deployment using the files replicated to the second scalable storage module.

Bootstrapping techniques for performing cross region disaster recovery

A data intake and query system can manage the search of large amounts of data using one or more processing nodes. The data intake and query system can identify a group of processing nodes and assign a first processing node of the group to download and search a particular data group based on a first node map. The data intake and query system may identify an action associated with the first processing node. The data intake and query system can cause a particular processing node of the group to download the particular data group based on a second node map and transmit an authorization to perform the action to the first processing node.

Deactivating a processing node based on assignment of a data group assigned to the processing node

Reassigning a processing node from downloading to searching a data group

Systems and methods are described for monitoring indexing nodes, populating and maintaining a resource catalog with relevant information, receiving requests for indexing node availability or assignments, identifying indexing nodes that are available to process data, and/or communicating information relating to available indexing nodes. The system can maintain the resource catalog based on communications with each of the containerized indexing nodes. The system can receive, from a partition manager of a data intake and query system, a request for a containerized indexing node that the partition manager can assign to process data received by the partition manager. The system can identify an available containerized indexing node to process the data. The system can communicate, to the partition manager, an indexing node identifier associated with the available containerized indexing node.

Identifying an indexing node to process data using a resource catalog

Systems and methods are disclosed for scalable bucket merging in a data intake and query system. Various components of a bucket manager can be used to monitor recently-created buckets of data in common storage that are associated with a particular tenant and a particular index, apply a comprehensive bucket merge policy to determine groups of buckets that qualify for merging, merge those group of buckets into merged buckets to be stored in the common storage, and update any information associated with the merged buckets and pre-merged buckets. These components may be shared across multiple tenants, and some of these components may be dynamically scalable based on need. This approach may also provide many additional benefits, including improved search performance from merged buckets, efficient resource utilization associated with discriminate merging, and redundancy in case of component failure.

Bucket merging for a data intake and query system using size thresholds

Systems and methods are disclosed for processing and executing queries in a data intake and query system. The data intake and query system receives raw machine data at an indexing system, and stores at least a portion of the raw machine data in buckets. Based on a determination that the size of multiple buckets satisfies a threshold size, the data intake and query system converts the buckets to non-editable buckets and stores the data in a remote shared storage system.

Managing and storing buckets to a remote shared storage system based on a collective bucket size

Reassigning data groups from backup to searching for a processing node

Systems and methods are disclosed for processing and executing queries in a data intake and query system. An indexing system of the data intake and query system receives data and stores at least a portion of it in buckets, which are then stored in a shared storage system. The indexing system merges multiple buckets to generate merged buckets and uploads the merged buckets to the shared storage system.

Merging buckets in a data intake and query system

Systems and methods are described for processing incoming data. The system can receive, from a first partition manager of a data intake and query system, first data that is associated with a first identifier, and can receive, from a second partition manager of the data intake and query system, second data that is associated with a second identifier. The system can process the first data and store first results of said processing the first data in one or more first buckets associated with the first tenant identifier. The system can process the second data and store second results of said processing the second data in one or more second buckets associated with the second tenant identifier.

Processing data associated with different tenant identifiers

Scalable bucket merging for a data intake and query system

Systems and methods are disclosed for processing and executing queries in a data intake and query system. An indexing system of the data intake and query system receives data from an ingestion buffer that includes a marker that indicates data that is made available to the indexing system. The data intake and query system stores at least a portion of the data in buckets and stores the buckets in a shared storage system. Based on the storage of the buckets in the shared storage system, the indexing system indicates to the ingestion buffer that the marker can be updated.

Updating a location marker of an ingestion buffer based on storing buckets in a shared storage system

Systems and methods are disclosed for interfacing with one or more components of a data intake and query system. The data intake and query system includes a gateway that interfaces between one or more computer-executable applications and one or more components of the data intake and query system. The data intake and query system can include an intake system configured to ingest data, an indexing system configured to generate and store one or more events based on the data, and a query system configured to execute one or more queries. The intake system can include a streaming data processor and at least one ingestion buffer. The indexing system can include at least one containerized indexing node, and the query system can include at least one containerized search node.

Data intake and query system gateway

Growing community of inventors

Union City, CA, United States of America

Sai Krishna Sajja