Splunk Inc.

Twitter, Inc.

Sanjeev Raghavendra Kulkarni

Boyang Peng

Karthikeyan Ramasamy

Poornima Devaraj

Jignesh Patel

Matteo Merli

Siddharth Taneja

Karthik Ramasamy

Maosong Fu

Sailesh Mittal

Nikunj Bhagat

Vikas Kedigehalli

Christopher Kellogg

Gaurav Chawla

Mehul Goyal

Zhimin Liang

Dragoljub Profirovic

Sumeet Rohatgi

Hao Zhan

Ivan Kelly

Aditya Kamble

Tyler Bakke

Rahul Merulinath

Ryan Flynn

Systems and methods are described for customizable data streams in a streaming data processing system. Routing criteria for the customizable data streams are defined by a user, an automated process, or any other process. The routing criteria can be defined using graphical controls. The streaming data processing system uses the routing criteria to determine data that should be used to populate a particular data stream. Further, processing pipelines are customized such that a particular processing pipeline can obtain data from a particular user defined data stream and write data to a particular user defined data stream. Data is routed through the user defined data streams and customized processing pipelines based on a data route. A data route for a set of data may include multiple user defined data streams and multiple processing pipelines. The data route can include a loop of processing pipelines and data streams.

User interface for customizing data streams and processing pipelines

A process for facilitating autoscaling in a stateful system is described herein. In embodiments, a set of metrics associated with a set of components of a stateful service is obtained. The set of metrics may generally indicate a utilization or a load of each of the components of the set of components (e.g., message managers and/or data stores). Thereafter, it is determined to initiate a scaling event at the stateful service in association with the set of components of the stateful service based on at least a portion of the set of metrics attaining a metric threshold indicating a threshold value for determining whether to scale stateful service components. A scaling request can then be provided to the stateful service to initiate the scaling event at the stateful service in association with the set of components of the stateful service.

Autoscaling in a stateful system

Systems and methods are described for implementation by a streaming data processing system of a processing pipeline that obtains data items from one or more data sources. The one or more data sources may include multiple partitions of a topic or multiple topics. The one or more data sources are defined by a data pattern, a group of tenants, or a process. The implementation of the processing pipeline can include identifying the one or more data sources to obtain data items. The processing pipeline can include a plurality of processing tasks to obtain the data items. The streaming data processing system can utilize a workload coordinator to dynamically assign particular data sources to particular processing tasks. The implementation of the processing pipeline can further include executing the plurality of processing tasks to cause processing tasks to obtain data items from assigned data sources and process the data items.

Workload coordinator for providing state credentials to processing tasks of a data processing pipeline

A process for ingesting raw machine data that reduces network and data intake and query system resources is described herein. For example, instead of routing the raw machine data to an intake ingestion buffer via a load balancer, a publisher may instead route metadata to the load balancer. The load balancer can use the metadata to identify an available virtual machine in the intake ingestion buffer. The load balancer can then provide to the publisher the public IP address of the available virtual machine. The publisher can communicate with the available virtual machine using the public IP address, and the available virtual machine can identify which virtual machine owns the topic related to the raw machine data. The publisher can then transmit raw machine data to the virtual machine that owns the topic.

Load balancer bypass for direct ingestion of data into a data intake and query system

User defined data stream for routing data to a data destination based on a data route

User interface for customizing data streams

A system and method for data stream processing. Two or more instances are connected as a topology, wherein at least one of the instances is a spout and at least one of the instances is a bolt. The topology is submitted to a scheduler, wherein the service scheduler receives resource offers from a cluster manager representing computing resources available on one or more of cluster nodes and determines resources to accept and computations to run on the accepted computing resources. The topology is scheduled as one or more jobs, wherein each job includes two or more containers, including a first container and a second container, the first container including a topology master and the second container including a stream manager and one or more stream processing system (SPS) instances, wherein each SPS instance represents one of the instances in the topology.

Stream processing at scale

Growing community of inventors

Los Altos, CA, United States of America

Sanjeev Raghavendra Kulkarni