Datatorrent, Inc.

Yahoo! Inc.

Amol J Kekre

Chetan Narsude

Thomas Weise

Phu D Hoang

Michael J Bennett

Hongyu Yao

Pramod Immaneni

David Yan

Gaurav Gupta

Ashwin Chandra Putta

Chandni Singh

Software for a distributed streaming platform receives a specification for a streaming application and converts the specification into a logical plan that includes a directed acyclic graph (DAG) with operators as nodes and streams as edges. The software translates the logical plan into a physical plan using any stream modes as specified and with one or more instances of the operators per any static partitioning. Then the software obtains a plurality of containers running on a networked cluster of servers. One slave executes in each container. The software assigns the instances to the slaves for execution according to an execution plan that depends at least in part on the physical plan and number of containers obtained. Each slave monitors throughput of the streaming windows through the instances in the container and other statistics related to service level and reports the results of the monitoring, which are used for dynamic adjustment.

Distributed streaming platform for real-time applications

Software receives a data stream for an application running on a distributed streaming platform over a networked cluster of servers. The software converts the data into a plurality of data tuples structured according to a schema. The software repeatedly emits a plurality of the data tuples as a streaming window, which is separated from other streaming windows by a leading control tuple associated with an ordinal identifier for the streaming window. The streaming window is a sequential sequence of tuples that is associated with a recovery policy. Then the software emits a checkpointing tuple after a plurality of streaming windows. The checkpointing tuple causes checkpointing of an instance of an operator for the application when the checkpointing tuple is received by the instance. Each of the operations is executed by one or more processors in real time or near real time rather than offline.

Checkpointing in distributed streaming platform for real-time applications

Software for a distributed streaming platform monitors a first operator instance in a container for an application running on a distributed streaming platform over a networked cluster of servers. The first operator instance includes an input port that receives streaming windows from an upstream instance and an output port that emits streaming windows to a downstream instance. And the monitoring includes determining a window identifier for a streaming window which the first operator instance is processing. The software reports results of the monitoring to a master and receives a dynamic adjustment from the master. The dynamic adjustment includes launching a second operator instance of the same operator in the container. The software resumes a flow of streaming windows into the input port and the output port according to first recovery policy for the first operator instance and a second recovery policy for the second operator instance.

Dynamic partitioning of instances in distributed streaming platform for real-time applications

Software for a distributed streaming platform receives an application. The application is structured as a directed acyclic graph (DAG) with instances of operators as nodes and streams as edges between nodes. Multiple instances of an operator access a shared database. The software receives a pre-defined hint associated with the application. The pre-defined hint sets a maximum period of time for local caching of a result from a query of the database by each of the multiple instances. The software launches the application by assigning the instances of operators to one or more containers provided by the streaming platform and initiating the streams. Each container is associated with a local cache. The software then receives a request from the application to make a dynamic adjustment that increases the maximum period of time for local caching of a result from a query of the database by each of the multiple instances.

Scalable local cache in distributed streaming platform for real-time applications

Software for a distributed streaming platform receives an application that runs on a streaming platform. The application is structured as a directed acyclic graph (DAG) with operators as nodes and streams as edges between nodes. The application includes an operator that receives a plurality of values, counts the values that are unique, and emits the unique values in a stream. The software partitions the operator into at least two partitioned instances. Then the software assigns a unique value emitting from a partitioned instance to one of a plurality of unifiers according to a pre-defined scheme. Each unifier creates a count of the unique values received by the unifier. Then the software transmits the counts from each of the unifiers to a downstream instance of an operator that aggregates the counts into a sum. And the software displays the sum in a graphical user interface (GUI).

Partitionable unifiers in distributed streaming platform for real-time applications

Software for a distributed streaming platform launches an application that runs on a streaming platform, in real time or near real time rather than offline. The application is structured as a directed acyclic graph (DAG) with instances of operators as nodes and streams as edges between nodes. The software receives an indication that a first instance of an operator is I/O bound. A stream connects the first instance of an operator to a second instance of another operator in a single container provided by the streaming platform. The software then transmits the indication to the application and receives a request to combine the first instance with the second instance into a single third instance of an operator. The software creates the third instance and re-initiates the stream using a recovery policy.

Thread-local streams in distributed streaming platform for real-time applications

Software for a distributed streaming platform receives an application that runs on a streaming platform. The application is structured as a directed acyclic graph (DAG) with instances of operators as nodes and streams as edges between nodes. The application is associated with a pre-defined hint that is a key-value pair. The software launches the application by assigning the instances of operators to containers provided by the streaming platform and initiating the streams. Then the software reads a value for the pre-defined hint and transmits the value to the application through an application programming interface (API) exposed by the streaming platform. The software receives a request from the application through the API to make a dynamic adjustment. And the software makes the dynamic adjustment and re-launches the application using a recovery policy.

Formula-based load evaluation in distributed streaming platform for real-time applications

Software for a distributed streaming platform receives a data stream for an application running on a distributed streaming platform over a networked cluster of servers. The software converts the data into a plurality of data tuples structured according to a schema. And the software repeatedly emits a specified number of the data tuples as a streaming window, which is separated from other streaming windows by a leading control tuple associated with an ordinal identifier for the streaming window and by a trailing control tuple associated with the same ordinal identifier. Then the software emits a checkpointing tuple following the trailing control tuple after a specified number of streaming windows. The checkpointing tuple causes checkpointing of an instance of an operator for the application when the checkpointing tuple is received by the instance.

A method for sampling of streaming data for distribution on a network (e.g., the Internet) includes receiving the steaming data (which includes periodically updated information items) from a streaming data source. The periodically updated information items of the streaming data are subsequently sampled using a sampling process based on a total update rate of the steaming data (U) and a capacity of the network (C), thereby creating a sampled data stream. Alternatively, the periodically updated information items of the streaming data can be sampled using a sampling process that is also based on the importance (K) of each of the periodically updated information items and the update rate (U) of each of the periodically updated information items to create the sampled data stream. The sampled data stream is then distributed on the network (e.g., the Internet or a private area network).

Method for the real-time distribution of streaming data on a network

A method of filtering real-time financial data to provide accurate data to users is provided. The method includes comparing the trading price of a currently reported stock trade with the trading price of a previously reported stock trade and identifies the currently reported stock trade as a potential bad trade if the percentage difference between the two is greater than a reference percentage. The real-time spike filter confirms the currently reported stock trade as a bad trade based on: (i) a comparison of the trading prices of the currently reported stock trade and a next reported stock trade; and (ii) a comparison of the trading prices of the previously reported stock trade and the next reported stock trade.

Spike filter for financial data represented as discrete-valued time series

A method for sampling of streaming data for distribution on a network (e.g., the Internet) includes receiving the streaming data (which includes periodically updated information items) from a streaming data source. The periodically updated information items of the streaming data are subsequently sampled using a sampling process based on a total update rate of the streaming data (U) and a capacity of the network (C), thereby creating a sampled data stream. Alternatively, the periodically updated information items of the streaming data can be sampled using a sampling process that is also based on the importance (K) of each of the periodically updated information items and the update rate (U) of each of the periodically updated information items to create the sampled data stream. The sampled data stream is then distributed on the network (e.g., the Internet or a private area network).

Growing community of inventors

Fremont, CA, United States of America

Amol J Kekre