Splunk Inc.

Cisco Technology, Inc.

Splunkinc.

Anish Shrigondekar

Ruochen Zhang

Zhenghui Xie

Shalabh Goyal

Alexandros Batsakis

Bharath Kishore Reddy Aleti

Bhavin Thaker

Mitchell Neuman Blank

Igor Stojanovski

Sai Krishna Sajja

Hasan Alayli

Rama Gopalan

Hongxun Liu

Jag Kerai

Ashish Mathew

Michael Lin

Mehul Goyal

Jagannath Kerai

Tianyi Gou

Sophia Rui Zhu

Douglas Rapp

Dhananjay Koshe

Eric Bond

Mitchell Blank, Jr

Eric Timothy Woo

Tameem Anwar

Aditya Dhoke

Megha Lakshminarayan

Steve Yu Zhang

Balaji Rao

Nitilaksha Halakatti

Nir Frenkel

Alexandras Batsakis

A data intake and query system can manage the search of large amounts of data using one or more processing nodes. When a new processing node is added or becomes available, the node coordinator can reassign duties from one or more processing nodes to the new processing node. The node coordinator can initially assign the new processing node one or more groups of data for backup purposes. At a later time, the node coordinator can reassign the new processing node to the one or more groups of data for searching purposes.

Reassignment of processing nodes

A data intake and query system can ingest and index large amounts of data using one or more ingestors and indexers. The ingestors can ingest incoming data, use it to generate events, and communicate the generated events to a message bus. Indexers can monitor their capacity to process additional groups of events. Based on a determination that an indexer has capacity to process one or more groups of events, the indexer can request one or more messages from the message bus, and process the events associated with the one or more messages.

Generating and storing aggregate data slices in a remote shared storage system

As an indexer indexes and groups events, it can generate data slices that include events. Based on a slice rollover policy, the indexer can add a particular slice to an aggregate slice. Based on an aggregate slice backup policy, the indexer can store a copy of the aggregate slice to a shared storage system. The aggregate slice can be used for restore purposes in the event the indexer fails or becomes unresponsive.

Indexing data at a data intake and query system based on a node capacity threshold

Resegmenting chunks of data for load balancing is disclosed. A plurality of first chunks of data is received. The plurality of first chunks of data includes one or more entries that include raw data produced by a component of an information technology environment and that reflects activity in the information technology environment. The plurality of first chunks of data is resegmented into a plurality of second chunks of data based on a source type of the plurality of first chunks. A first subset of the plurality of second chunks of data is distributed to a first indexer of a set of indexers. An occurrence of a trigger event is determined, and in response to the trigger event, a second subset of the plurality of second chunks of data is distributed to a second indexer of the set of indexers.

Resegmenting chunks of data based on one or more criteria to facilitate load balancing

A data intake and query system can generate local data enrichment objects and receive federated data enrichment objects from another data intake and query system. In response to receiving a query, the data intake and query system can determine whether the query is subquery of a federated query. If the query is a subquery, the data intake and query system can use the federated data enrichment objects to execute the query.

Federated data enrichment objects

Reassigning a processing node from downloading to searching a data group

An actionable event collector in a server cluster receives information specifying an actionable event instance regarding an actionable event occurrence in the server cluster. The actionable event collector transmits a representation of the actionable event instance to an actionable event queue builder. The actionable event queue builder inserts the representation as an entry into an actionable event queue. The event action dispatcher processes the entry from the actionable event queue, wherein processing the entry comprises determining a responsive action for the entry and causing performance of the responsive action.

Actionable event responder architecture

A computer implemented system is described for assigning executable jobs to pipeline sets, whereby the jobs may be network based computer jobs. The assigning includes generating a weight for each pipeline set of multiple pipeline sets to obtain multiple weights. Generating a weight includes obtaining duty cycle metrics for pipeline software threads in the pipeline set. The duty cycle metrics include a measure of an amount of time that a corresponding pipeline thread is executing and actively processing data. Generating the weight further includes determining the weight for the pipeline set based at least in part on the duty cycle metrics. The method further includes assigning a job request to a target pipeline set selected from the pipeline sets according to a weighted random algorithm, wherein the weighted random algorithm uses the weights.

Duty cycle estimation for job assignment

Resource management includes storing, for multiple workload pools of a data intake and query system, a workload pool hierarchy arranged in multiple workload pool layers. After storing a processing request is assigned a selected subset of workload pools in a second layer of the workload pool hierarchy based on a type of processing request. The processing request is then assigned to an individual workload pool in the selected subset to obtain a selected workload pool. Execution of the processing request is initiated on the selected workload pool.

Workload pool hierarchy for a search and indexing system

Cascading payload replication to target compute nodes is disclosed. Cascading payload replication can be accomplished using a two-stage operation for a replication operation. In the first stage, a plan is generated and distributed for the replication operation. The plan includes an assignment of compute nodes to tree nodes in a tree hierarchy. In the second phase, the payload is distributed according to the plan. The plan is different for at least two replication operations. Thus, the cascading payload replication is adaptable to changing target compute nodes and provides for load balancing.

Cascading payload replication

Dynamic reassignment of search processes into workload pools includes receiving a search query to search at least one data store, assigning the search query to a first workload pool, and executing the search query using a first hardware resource in the first workload pool, the first hardware resource corresponding to a first portion of a hardware device. Dynamic reassignment further includes receiving, while executing the search query, an update command to move the search query to a second workload pool, moving, while executing the search query, the search query to the second workload pool; and continuing execution of the search query using a second hardware resource in the second workload pool. The second hardware resource corresponds to a second portion of the hardware device.

Dynamic reassignment in a search and indexing system

A data intake and query system can ingest and index large amounts of data using one or more ingestors and indexers. The ingestors can ingest incoming data and use it to generate events. The ingestor can group the events and prepare them for communication to a message bus. The ingestor can determine a size of the group of events. If the size of the group of events satisfies a message size threshold, the ingestor can store the group of events to a data store, obtain a reference to the group of events, and communicate the reference to the group of events to a message queue. An indexer can obtained the reference from the message queue and use the reference to obtain the group of events from the data store.

Using a data store and message queue to ingest data for a data intake and query system

Reassigning data groups from backup to searching for a processing node

Resegmenting chunks of data based on source type to facilitate load balancing

Growing community of inventors

Sunnyvale, CA, United States of America

Anish Shrigondekar