Splunk Inc.

Splunkinc.

Bharath Kishore Reddy Aleti

Alexandros Batsakis

Sourav Pal

Arindam Bhattacharjee

Christopher Madden Pride

Ashish Mathew

James Monschke

Anish Shrigondekar

Mitchell Neuman Blank

Amritpal Singh Bath

Rama Gopalan

Hongxun Liu

Clara Elizabeth Lee

Tingjin Xu

Kartheek Babu Kolla

Chaithra Nataraj

Octavio Enrique Di Sciullo

Jason Andrew Beyers

Sai Krishna Sajja

Alexander Douglas James

Igor Stojanovski

Eric Sammer

Joseph Gabriel Echeverria

Karthikeyan Sabhanatarajan

Tameem Anwar

Yuan Xu

Ankit Jain

Steve Wong

Manu Jose, Jr

Eric Woo

Alexandras Batsakis

Paul J Lucas

Systems and methods are disclosed for processing and executing queries in a data intake and query system. The data intake and query system receives raw machine data at an indexing system, and stores at least a portion of the raw machine data in buckets using containerized indexing nodes instantiated in a containerized environment. The data intake and query system stores the buckets in a shared storage system.

Processing data using nodes in a scalable environment

Processing data using containerized nodes in a containerized scalable environment

Systems and methods are disclosed for processing and executing queries in a data intake and query system. The data intake and query system receives a query identifying a set of data to be processed and a manner of processing the set of data. The data intake and query system identifies buckets that are to be searched and search nodes to execute the query. The data intake and query system maps the identified buckets to the search nodes and executes the query using the identified bucket and search nodes.

Managing efficient query execution including mapping of buckets to search nodes

Systems and methods are disclosed for processing and executing queries in a data intake and query system. The data intake and query system receives a query identifying a set of data to be processed and a manner of processing the set of data. The data intake and query system uses a search node catalog to identify search nodes that are available to execute the query and uses a bucket catalog to identify buckets to be searched. The data intake and query system executes the query using the identified bucket and search nodes.

Identifying buckets for query execution using a catalog of buckets

Systems and methods are disclosed for monitoring features of a computing device of a distributed computing system using a self-monitoring module. The self-monitoring module can include multiple feature-specific monitoring modules and one or more parent nodes for the feature-specific monitoring modules. A feature-specific monitoring module can identify or detect a fault status change, such as a fault condition or fault resolution, for one or more features. Based on the identified fault conditions or fault resolutions, the feature-specific monitoring module can determine an internal status and communicate an updated status to a parent node.

Monitoring statuses of monitoring modules of a distributed computing system

Embodiments of the present disclosure provide solutions for determining an elected search head captain is unqualified for the position, identifying a more qualified search head, and transferring the captain position to the more qualified search head. A method is provided that includes referencing qualification parameters in an elected search head captain, determining whether the newly elected search head captain is qualified for the position based on the parameters, identifying a more qualified search head to be the search head captain if the newly elected search head captain is determined to be unqualified for the position, and transferring the position of captain to the more qualified search head. The qualification parameters may include, for example, a pre-determined static flag set by an administrator of the search environment, and configuration replication status that corresponds to the most recent configuration state of the search head as recorded by the previous search head captain.

Qualification parameters for captain selection in a search head cluster

A computer implemented system is described for assigning executable jobs to pipeline sets, whereby the jobs may be network based computer jobs. The assigning includes generating a weight for each pipeline set of multiple pipeline sets to obtain multiple weights. Generating a weight includes obtaining duty cycle metrics for pipeline software threads in the pipeline set. The duty cycle metrics include a measure of an amount of time that a corresponding pipeline thread is executing and actively processing data. Generating the weight further includes determining the weight for the pipeline set based at least in part on the duty cycle metrics. The method further includes assigning a job request to a target pipeline set selected from the pipeline sets according to a weighted random algorithm, wherein the weighted random algorithm uses the weights.

Duty cycle estimation for job assignment

Resource management includes storing, for multiple workload pools of a data intake and query system, a workload pool hierarchy arranged in multiple workload pool layers. After storing a processing request is assigned a selected subset of workload pools in a second layer of the workload pool hierarchy based on a type of processing request. The processing request is then assigned to an individual workload pool in the selected subset to obtain a selected workload pool. Execution of the processing request is initiated on the selected workload pool.

Workload pool hierarchy for a search and indexing system

Dynamic reassignment of search processes into workload pools includes receiving a search query to search at least one data store, assigning the search query to a first workload pool, and executing the search query using a first hardware resource in the first workload pool, the first hardware resource corresponding to a first portion of a hardware device. Dynamic reassignment further includes receiving, while executing the search query, an update command to move the search query to a second workload pool, moving, while executing the search query, the search query to the second workload pool; and continuing execution of the search query using a second hardware resource in the second workload pool. The second hardware resource corresponds to a second portion of the hardware device.

Dynamic reassignment in a search and indexing system

Systems and methods are disclosed for processing and executing queries in a data intake and query system. The data intake and query system receives a query identifying a set of data to be processed and a manner of processing the set of data. The data intake and query system identifies buckets that are to be searched and stores a copy of buckets in memory associated with one or more search nodes. A search node performs a search on buckets residing in its memory.

Copying buckets from a remote shared storage system to memory associated with a search node for query execution

Systems and methods are disclosed for processing and executing queries in a data intake and query system. The data intake and query system receives a query identifying a set of data to be processed and a manner of processing the set of data. The data intake and query system identifies buckets that are to be searched. The data intake and query system performs a hash on bucket identifiers of the identified buckets to identify search nodes to search the buckets.

Hashing bucket identifiers to identify search nodes for efficient query execution

Pipeline set selection based on duty cycle estimation

Achieving search and ingest isolation via resource management in a search and indexing system includes receiving a search query associated with at least one data store, assigning, in response to the search query being associated with the at least one data store, the search query to a first workload pool in a set of query workload pools, and processing the search query using a first hardware resource in the first workload pool. Achieving search and ingest isolation further includes receiving an ingest request comprising data associated with the at least one data store. The ingest request is assigned to a second workload pool in a set of ingest workload pools. The set of query workload pools and the set of ingest workload pools are disjoint. Achieving search and ingest isolation further includes processing the ingest request using a second hardware resource in the second workload pool.

Achieve search and ingest isolation via resource management in a search and indexing system

Monitoring features of components of a distributed computing system

Systems and methods are disclosed for processing and executing queries in a data intake and query system. The query identifies a set of data to be processed and a manner of processing the set of data. The data intake and query system dynamically identifies a plurality of containerized search nodes instantiated on one or more computing devices in a containerized environment to execute the query. The data intake and query system executes the query using the containerized search nodes.

Query execution using containerized state-free search nodes in a containerized scalable environment

Growing community of inventors

San Ramon, CA, United States of America

Bharath Kishore Reddy Aleti