Twitter, Inc.

Karthik Ramasamy

Maosong Fu

Jignesh Patel

Sanjeev Raghavendra Kulkarni

Siddharth Taneja

Sailesh Mittal

Nikunj Bhagat

Vikas Kedigehalli

Christopher Kellogg

Cong Wang

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for data stream processing. One method performed by a stream manager in a group of containers including streaming processes and stream managers includes receiving a first data streaming object of the data streaming objects from a first component in the group of containers; determining whether to process the first data streaming object; in response to determining to process the first data streaming object processing the first data streaming object and routing the first data streaming object; and in response to determining not to process the first data streaming object, routing the first data streaming object.

Real-time analysis of data streaming objects for distributed stream processing

A system and method for data stream processing. Two or more instances are connected as a topology, wherein at least one of the instances is a spout and at least one of the instances is a bolt. The topology is submitted to a scheduler, wherein the service scheduler receives resource offers from a cluster manager representing computing resources available on one or more of cluster nodes and determines resources to accept and computations to run on the accepted computing resources. The topology is scheduled as one or more jobs, wherein each job includes two or more containers, including a first container and a second container, the first container including a topology master and the second container including a stream manager and one or more stream processing system (SPS) instances, wherein each SPS instance represents one of the instances in the topology.

Stream processing at scale

Growing community of inventors

San Francisco, CA, United States of America

Karthik Ramasamy