International Business Machines Corporation

Daniel Beuch

John Matthew Santosuosso

Eric L Barsness

Michael J Branson

Alexander Cook

Robert J Bestgen

Brian Robert Muras

Ryan Kirk Cradick

Adam Thomas Stallman

Shantan Kethireddy

Mark John Anderson

Michael D Pfeifer

Randy William Ruhlow

Scott Forstie

Thomas Owen McKinley

James Michael Flanagan

Paul S Nelsestuen

Curtis Neal Boger

Theresa Renee Euler

Michael K Branson

A stream of tuples is processed by a stream application. The stream application includes a plurality of processing elements that operate on one or more compute nodes, each processing element includes one or more stream operators. One or more databases that are capable of communicating with the stream application are monitored during the processing of the stream of tuples. A potential performance condition of a first database of the one or more databases is detected based on the monitoring. An output adjustment is performed, in response to the potential performance condition.

Datastore aware streams computing

An embodiment includes analyzing data associated with an original flow graph comprising a plurality of operators of a stream computing application, including identifying a secure network connection between a first operator and a second operator that uses encryption. The embodiment fuses the first operator with the second operator such that a first logical function of the first operator is combined with a second logical function of the second operator. The embodiment then generates a modified flow graph as a modification of the original flow graph that combines the first operator and the second operator and lacks encryption between the first operator and the second operator.

Using fusion to reduce encryption in streaming applications

Feedback tuples are implemented in a stream computing environment to provide data flow status information for use in detecting, identifying and/or handling data flow anomalies for a stream computing application.

Feedback tuples for detecting data flow anomalies in stream computing environment

Queries are monitored in a database which receives input from a stream computing application to identify data of interest. Parameters defining the data of interest, which are preferably expressed as a logical query, are sent to the stream computing application, which then processes the in-flight streamed data satisfying the parameters in some special manner. In some embodiments, the stream computing application increases the processing priority of in-flight data satisfying the parameters. In some embodiments, the stream computing application applies additional processing steps to the in-flight data satisfying the parameters to provide enhanced data or metadata.

Managing the processing of streamed data in a data streaming application using query information from a relational database

An embodiment includes monitoring a distributed computing application at runtime for occurrence of a condition, where the condition includes occurrence of a degradation condition on a processing element (PE). The embodiment also includes starting a provisional PE on a second node at runtime while maintaining runtime operation of the PE. The embodiment compares operation of the PE to operation of the provisional PE based on a performance metric associated with the computing resource. The embodiment connects the provisional PE to replace the stream of tuples from the PE to a downstream PE with a stream of tuples from the provisional PE to the downstream PE.

Dynamic replacement of degrading processing elements in streaming applications

Certain aspects of the present disclosure provide techniques for identifying temporal data in data streams to create a temporal database for a stream(s) application to query for temporal data. An example technique includes receiving streams of data at a streams engine and processing the streams of data according to a priority order. The streams engine identifies whether the database is a temporal database and identifies temporal data in each stream of data based on frame analysis, natural language processing techniques, metadata, and optical character recognition. Further, the streams engine generates context data corresponding to the temporal data. The streams engine generates a temporal data record based on the temporal data and context data, and the streams engine generates a reliability factor. The temporal data record and reliability factor are stored in the temporal database for a stream application to query regarding temporal information at a later point in time.

Integrating relational database temporal tables with a distributed programming environment

Profiling data characterizing a data streaming application is used to predict data which will need to be retrieved by a processing element during execution of the data streaming application. Data is retrieved responsive to the prediction, in advance of actual demand by the processing element which requires it. Prediction may be based at least in part on upstream tuple contents, and could include other historical data retrieval patterns. In some embodiments, retrieval of predicted data may be delayed so that data is retrieved just in time.

Reducing flow delays in a data streaming application caused by lookup operations

A method, system and computer program product for staggering a stream application's deployment, by: providing one or more configuration settings that define one or more delays for instantiation or initialization of at least one target processing element of the stream application based on the stream application's run-time conditions or events; and instantiating or initializing the target processing element of the stream application when the delays defined by the configuration settings have been satisfied. The configuration settings may define the delays to wait until a specified period of time has lapsed after one or more other processing elements have been instantiated or initialized, or to wait until a specified amount of data has been processed by one or more other processing elements, or to wait until one or more signals is received from one or more other processing elements, before instantiating or initializing the target processing element.

Staggering a stream application's deployment

Disclosed aspects relate to window management in a stream computing environment. A set of computing resources may be detected with respect to the stream computing environment. Based on the set of computing resources, a set of window configurations in the stream computing environment may be determined. In response to determining the set of window configurations in the stream computing environment, the set of window configurations may be established in the stream computing environment.

Window management based on a set of computing resources in a stream computing environment

A computer system controls processing elements associated with a stream computing application. A stream computing application is monitored for the occurrence of one or more conditions. One or more processing element groups are determined to be restarted based on occurrence of the one or more conditions, wherein the processing element groups each include a plurality of processing elements associated with the stream computing application. Each processing element of the determined one or more processing element groups is concurrently restarted. Embodiments of the present invention further include a method and program product for controlling processing elements within a stream computing application in substantially the same manner described above.

Controlling processing elements in a distributed computing environment

A database which receives input from a data streaming application may include in-flight streamed data within the scope of a command which alters database data. Preferably, the streaming application produces data tuples for input to at least one table of the database. A user submitting modify data commands to apply to multiple tuples meeting some specified logical parameters may define the command scope to include in-flight data in the streaming application, causing the database manager to forward the command to the streaming application. A streaming application manager invokes database agents in respective nodes of the streaming application, which identify in-flight data meeting the specified logical parameters of the command, and apply the modify data operation to such data.

Altering in-flight streamed data from a relational database

The present invention provides a computer implemented method, system, and computer program product of assigning processing elements to stream computing servers with respect to external computing resources. In an embodiment, the present invention includes receiving performance requirements data of a plurality of processing elements of a computer software application, where the processing elements are configured to execute on a stream computing system, receiving resource characteristics data of at least one computing resource external to the stream computing system, receiving performance characteristics data of stream computing servers, and in response to receiving a request from at least one processing element to communicate with the at least one computing resource external to the stream computing system, assigning the at least one processing element to at least one stream computing server as a function of the performance requirements data, the resource characteristics data, and the performance characteristics data.

Assigning processing elements to stream computing servers

Systems, methods, and articles of manufacture are disclosed for processing element (PE) deployment placement in a streaming data system. In particular, PEs of a stream computing application are grouped across compute nodes in a dynamic manner that improves over manually specified groupings of PEs on particular compute nodes. A stream manager deploys PEs according to rules in deployment instructions specifying whether to collocate PEs on a single compute node and/or ex-collocate PEs across compute nodes based on dynamically-determined information relating to the PEs themselves, compute nodes on which PEs are to run, and/or the streams runtime. For example, the stream manager may profile code and deploy PEs on compute nodes based on such profiling, deploy PEs together on compute nodes meeting predefined system characteristics or streams runtime conditions or metrics, and/or deploy PEs on a compute node within a predefined proximity of an external source.

Growing community of inventors

Rochester, MN, United States of America

Daniel Beuch