Splunk Inc.

James Monschke

Alexandros Batsakis

Ashish Mathew

Christopher Madden Pride

Sourav Pal

Arindam Bhattacharjee

Bharath Kishore Reddy Aleti

Alexander Douglas James

Xiaowei Wang

Scott Calvert

Bei Li

Sogol Moshtaghi

Igor Stojanovski

Sai Krishna Sajja

Karthikeyan Sabhanatarajan

Tameem Anwar

Steve Wong

Paul J Lucas

Systems and methods are disclosed for processing and executing queries in a data intake and query system. The data intake and query system receives raw machine data at an indexing system, and stores at least a portion of the raw machine data in buckets using containerized indexing nodes instantiated in a containerized environment. The data intake and query system stores the buckets in a shared storage system.

Processing data using nodes in a scalable environment

Processing data using containerized nodes in a containerized scalable environment

Systems and methods are disclosed for processing and executing queries in a data intake and query system. The data intake and query system receives a query identifying a set of data to be processed and a manner of processing the set of data. The data intake and query system identifies buckets that are to be searched and search nodes to execute the query. The data intake and query system maps the identified buckets to the search nodes and executes the query using the identified bucket and search nodes.

Managing efficient query execution including mapping of buckets to search nodes

Systems and methods are disclosed for processing and executing queries in a data intake and query system. The data intake and query system receives a query identifying a set of data to be processed and a manner of processing the set of data. The data intake and query system uses a search node catalog to identify search nodes that are available to execute the query and uses a bucket catalog to identify buckets to be searched. The data intake and query system executes the query using the identified bucket and search nodes.

Identifying buckets for query execution using a catalog of buckets

Systems and methods are disclosed for mapping search nodes to a search head in a data intake and query system based on a tenant identifier in order to execute a query received by the data intake and query system. The mapping may allow same or similar search nodes to be used to execute queries that are associated with a particular tenant identifier, in order to take advantage of caching and local data stored with those search nodes. In some cases, search nodes can be mapped based on the tenant identifier using a hashing algorithm, such as a consistent hashing algorithm.

Dynamically assigning a search head to process a query

Systems and methods are disclosed for processing and executing queries in a data intake and query system. The data intake and query system receives a query identifying a set of data to be processed and a manner of processing the set of data. The data intake and query system identifies buckets that are to be searched and stores a copy of buckets in memory associated with one or more search nodes. A search node performs a search on buckets residing in its memory.

Copying buckets from a remote shared storage system to memory associated with a search node for query execution

Systems and methods are disclosed for processing and executing queries in a data intake and query system. The data intake and query system receives a query identifying a set of data to be processed and a manner of processing the set of data. The data intake and query system identifies buckets that are to be searched. The data intake and query system performs a hash on bucket identifiers of the identified buckets to identify search nodes to search the buckets.

Hashing bucket identifiers to identify search nodes for efficient query execution

Mapping search nodes to a search head using a tenant identifier

Systems and methods are disclosed for dynamically assigning a search head or search nodes in a data intake and query system for a query received by the data intake and query system. Existing search heads and search nodes can periodically report their status to the data intake and query system, which can use that information to help determine the need to provision additional search heads and search nodes. The data intake and query system can receive a query identifying a set of data to be processed and a manner of processing the set of data. The data intake and query system can use the status information for existing search heads and search nodes to dynamically assign a search head and search nodes for the query. Dynamically assigning the search head and search nodes in this manner may provide many benefits, including improved load balancing and resource utilization.

Dynamically assigning a search head and search nodes for a query

Systems and methods are disclosed for processing and executing queries in a data intake and query system. The query identifies a set of data to be processed and a manner of processing the set of data. The data intake and query system dynamically identifies a plurality of containerized search nodes instantiated on one or more computing devices in a containerized environment to execute the query. The data intake and query system executes the query using the containerized search nodes.

Query execution using containerized state-free search nodes in a containerized scalable environment

Search node and bucket identification using a search node catalog and a data store catalog

Systems and methods are disclosed for processing and executing queries in a data intake and query system. The data intake and query system maintains a catalog of buckets stored in a remote shared storage system. The buckets store raw machine data associated with a timestamp. The data intake and query receives a query identifying a set of data to be processed and a manner of processing the set of data, and executes the query based on the catalog of buckets.

Identifying buckets for query execution using a catalog of buckets stored in a remote shared storage system

Systems and methods are disclosed for processing and executing queries in a data intake and query system. The data intake and query system receives a query identifying a set of data to be processed and a manner of processing the set of data. The data intake and query system uses one or more containerized search nodes to execute the query and stores the results in a data store for combination with additional query results.

Growing community of inventors

San Francisco, CA, United States of America

James Monschke