Splunk Inc.

Asha Andrade

Sourav Pal

Arindam Bhattacharjee

Xiaowei Wang

Vanco Buca

Nikhil Roy

Anuradha Pariti

Tingting Bao

Weichao Duan

The disclosed implementations include a method performed by a data intake and query system. The method includes receiving a search query at a search head, the search query including a branching operation between sets of data, generating a first subquery and a second subquery corresponding to the sets of data for execution by a search node, generating instructions for an intermediary node to combine partial results of the first subquery and the second subquery and instructions to concurrently communicate the subqueries to a search node, and executing the query by providing the instructions for the intermediary node to the intermediary node and the subqueries to the search node, the intermediary node receiving sets of partial search results for the subqueries, performing at least a portion of the branching operation on the partial results, and communicating the combined results to another intermediary node or the search head.

Parallel branch operation using intermediary nodes

Systems and methods are described for determining a quantity of records generated by a processing task of a query executed in a data intake and query. The system receives a query and identifies a processing task of the query and a quantity of records to be processed according to the query. The system determines the number of records generated by the processing task based on the number of records to be processed and a record generation estimate. The system can allocate compute resources or determine a query execution time for at least a portion of the query based on the determined quantity of records generated.

Determining records generated by a processing task of a query

Systems and methods are described for exporting bucket data from one or more buckets to one or more worker nodes. The system can identify data from different bucket data from buckets stored in a data intake and query system that is to be processed by one or more worker nodes. The system can allocate one or more execution resources, such as a processing pipeline, to process and export the bucket data from the buckets. The system can assign bucket data corresponding to individual buckets to the execution resource based on a bucket distribution policy. The indexer can export the bucket data to the worker nodes for further processing based on the bucket data-execution resource assignment.

Bucket data distribution for exporting data to worker nodes

Systems and methods are described for determining a record generation estimate related to a particular processing task. The system obtains a sample set of data that includes multiple records. The system applies a processing task, such as a transform or regular expression rule to the sample set of data and determines how many records are generated by the processing task. Based on the number of records generated, the system determines a record generation estimate. The system can use the record generation estimate to allocate compute resources or determine a query execution time for at least a portion of the query based on the record generation estimate.

Determining a record generation estimate of a processing task

Systems and methods for a data index and query system that utilize a query acceleration data store. An example method includes receiving a query identifying a set of data to be processed and a manner of processing the set of data. A query processing scheme for obtaining and processing the set of data is defined. First partial results of the query stored in a data store are identified, with the first partial results corresponding to a first portion of the set of data. One or more partitions are dynamically allocated to obtain a second portion of the set of data from different data sources. The second portion of the set of data is processed to obtain second partial results. The first partial results and second partial results are combined. The query is executed based on the query processing scheme.

Growing community of inventors

Sunnyvale, CA, United States of America

Asha Andrade