Splunk Inc.

Cisco Technology, Inc.

Tianyi Gou

Sai Krishna Sajja

Alexandros Batsakis

Igor Stojanovski

Zhenghui Xie

Tameem Anwar

Ashish Mathew

Anish Shrigondekar

Ruochen Zhang

Mehul Goyal

Sophia Rui Zhu

Douglas Rapp

Eric Timothy Woo

Jiahan Wang

Kartheek Babu Kolla

Abhinav Prasad Nekkanti

Kai-Sern Lim

Srinivas Bobba

Srinivas Chowdhary Bobba

A data intake and query system can manage the search of large amounts of data using one or more processing nodes. When a new processing node is added or becomes available, the node coordinator can reassign duties from one or more processing nodes to the new processing node. The node coordinator can initially assign the new processing node one or more groups of data for backup purposes. At a later time, the node coordinator can reassign the new processing node to the one or more groups of data for searching purposes.

Reassignment of processing nodes

A data intake and query system can manage the search of large amounts of data using one or more processing nodes. The data intake and query system can identify a first group of processing nodes and cause a first processing node of the group to download and search a particular data group based on a first node map. The data intake and query system may identify a second group of processing nodes that includes the first group of processing nodes and a second processing node. The data intake and query system can transmit commands to cancel one or more data group downloads at the first processing node and can reassign the corresponding data groups for download to the second processing node based on a second node map.

Reassigning data groups based on activation of a processing node

A data intake and query system can manage the search of large amounts of data using one or more processing nodes. The data intake and query system can identify a group of processing nodes and assign a first processing node of the group to download and search a particular data group based on a first node map. The data intake and query system may identify an action associated with the first processing node. The data intake and query system can cause a particular processing node of the group to download the particular data group based on a second node map and transmit an authorization to perform the action to the first processing node.

Deactivating a processing node based on assignment of a data group assigned to the processing node

Reassigning a processing node from downloading to searching a data group

Systems and methods are disclosed for scalable bucket merging in a data intake and query system. Various components of a bucket manager can be used to monitor recently-created buckets of data in common storage that are associated with a particular tenant and a particular index, apply a comprehensive bucket merge policy to determine groups of buckets that qualify for merging, merge those group of buckets into merged buckets to be stored in the common storage, and update any information associated with the merged buckets and pre-merged buckets. These components may be shared across multiple tenants, and some of these components may be dynamically scalable based on need. This approach may also provide many additional benefits, including improved search performance from merged buckets, efficient resource utilization associated with discriminate merging, and redundancy in case of component failure.

Bucket merging for a data intake and query system using size thresholds

Reassigning data groups from backup to searching for a processing node

Scalable bucket merging for a data intake and query system

Growing community of inventors

San Francisco, CA, United States of America

Tianyi Gou