Datatorrent, Inc.

Phu D Hoang

Chetan Narsude

Amol J Kekre

Thomas Weise

Software for a distributed streaming platform receives a specification for a streaming application and converts the specification into a logical plan that includes a directed acyclic graph (DAG) with operators as nodes and streams as edges. The software translates the logical plan into a physical plan using any stream modes as specified and with one or more instances of the operators per any static partitioning. Then the software obtains a plurality of containers running on a networked cluster of servers. One slave executes in each container. The software assigns the instances to the slaves for execution according to an execution plan that depends at least in part on the physical plan and number of containers obtained. Each slave monitors throughput of the streaming windows through the instances in the container and other statistics related to service level and reports the results of the monitoring, which are used for dynamic adjustment.

Distributed streaming platform for real-time applications

Software receives a data stream for an application running on a distributed streaming platform over a networked cluster of servers. The software converts the data into a plurality of data tuples structured according to a schema. The software repeatedly emits a plurality of the data tuples as a streaming window, which is separated from other streaming windows by a leading control tuple associated with an ordinal identifier for the streaming window. The streaming window is a sequential sequence of tuples that is associated with a recovery policy. Then the software emits a checkpointing tuple after a plurality of streaming windows. The checkpointing tuple causes checkpointing of an instance of an operator for the application when the checkpointing tuple is received by the instance. Each of the operations is executed by one or more processors in real time or near real time rather than offline.

Checkpointing in distributed streaming platform for real-time applications

Software for a distributed streaming platform monitors a first operator instance in a container for an application running on a distributed streaming platform over a networked cluster of servers. The first operator instance includes an input port that receives streaming windows from an upstream instance and an output port that emits streaming windows to a downstream instance. And the monitoring includes determining a window identifier for a streaming window which the first operator instance is processing. The software reports results of the monitoring to a master and receives a dynamic adjustment from the master. The dynamic adjustment includes launching a second operator instance of the same operator in the container. The software resumes a flow of streaming windows into the input port and the output port according to first recovery policy for the first operator instance and a second recovery policy for the second operator instance.

Dynamic partitioning of instances in distributed streaming platform for real-time applications

Software for a distributed streaming platform receives a data stream for an application running on a distributed streaming platform over a networked cluster of servers. The software converts the data into a plurality of data tuples structured according to a schema. And the software repeatedly emits a specified number of the data tuples as a streaming window, which is separated from other streaming windows by a leading control tuple associated with an ordinal identifier for the streaming window and by a trailing control tuple associated with the same ordinal identifier. Then the software emits a checkpointing tuple following the trailing control tuple after a specified number of streaming windows. The checkpointing tuple causes checkpointing of an instance of an operator for the application when the checkpointing tuple is received by the instance.

Growing community of inventors

Portola Valley, CA, United States of America

Phu D Hoang