International Business Machines Corporation

Alexander Cook

John Matthew Santosuosso

Eric L Barsness

Manuel Orozco

Jason Allan Nikolai

David M Koster

Christopher R Sabotta

Daniel Beuch

Cory J Kleinheksel

Michael J Branson

Peter Andrew Nicholls

Alexander J Pogue

Brian Robert Muras

Jonathan Louis Kaus

Ryan Kirk Cradick

Joseph C Schmidt

James E Carey

A method, computer program product, and computing device for monitoring network activity associated with streaming a data load through a stream application including a plurality of stream operators deployed on a plurality of computing devices. One or more stream operators with one or more external connections may be identified from the plurality of stream operators. The identified one or more stream operators may be deployed based upon, at least in part, the one or more external connections.

System and method for identifying external connections in a streaming application

Queries are monitored in a database which receives input from a stream computing application to identify data of interest. Parameters defining the data of interest, which are preferably expressed as a logical query, are sent to the stream computing application, which then processes the in-flight streamed data satisfying the parameters in some special manner. In some embodiments, the stream computing application increases the processing priority of in-flight data satisfying the parameters. In some embodiments, the stream computing application applies additional processing steps to the in-flight data satisfying the parameters to provide enhanced data or metadata.

Managing the processing of streamed data in a data streaming application using query information from a relational database

A connection can be made to a processing element of a remotely deployed and live streaming application executed by a first data processing system, the processing element containing at least one operator that processes at least one tuple. As the live streaming application is executed, without slowing or modifying data flow of the live streaming application execution to client devices, a copy of the tuple and a memory dump of state data for a state of the operator can be received, and the tuple can be tracked through a call graph. The state data can be loaded into a local instance of the operator loaded into a debugger. At least a portion of the call graph can be presented to a user, and a flow of the tuple through the call graph based on the state data for the operator can be indicated.

Debugging a live streaming application

A method, computer program product, and computer system for identifying, in a dynamically formed cloud, a plurality of voluntary compute nodes, wherein the plurality of voluntary compute nodes may include personal computing devices of a plurality of users, and wherein at least a portion of the plurality of voluntary compute nodes may be configured to communicate with each other to receive and send streaming data. A first compute node of at least the portion of the plurality of voluntary compute nodes may be identified to receive and distribute a streams operator of a streams graph requested by a client computing device. A second compute node of at least the portion of the plurality of voluntary compute nodes may be identified for verifying computational correctness of a tuple stream computation for the stream operator by replicating the tuple stream computations for the stream operator on the second compute node. A final result tuples of the graph may be sent to the client computing device.

System and method for dynamic cloud formations for stream computing

System resource usage by a streaming application processing workloads can be monitored, the streaming application comprising at least one parallelizable operator, at least a first instance of the parallelizable operator being initialized to execute. Based on the monitoring, at least one performance metric for the streaming application in processing the workloads can be determined. Based on the at least one performance metric, whether hosting of the streaming application is, or is near to, violating at least one criteria can be determined. If so, at least one additional instance of at least one parallelizable operator of the streaming application that is currently executing can be initialized. Responsive to initializing the at least one additional instance of at least one of the parallelizable operators, a portion of the workloads can be directed to the at least one additional instance of at least one of the parallelizable operators.

Optimizing streaming graph topology based on service level agreement

Disclosed aspects relate to cache management in a stream computing environment that uses a set of many-core hardware processors to process a stream of tuples by a plurality of processing elements which operate on the set of many-core hardware processors. The stream of tuples to be processed by the plurality of processing elements which operate on the set of many-core hardware processors may be received. A tuple-processing hardware-route on the set of many-core hardware processors may be determined based on a cache factor associated with the set of many-core hardware processors. The stream of tuples may be routed based on the tuple-processing hardware-route on the set of many-core hardware processors. The stream of tuples may be processed by the plurality of processing elements which operate on the set of many-core hardware processors.

Cache management in a stream computing environment that uses a set of many-core hardware processors

Aspects of the disclosure relate to using a set of warning data to manage distributed development of an application in a computing environment. In embodiments, the computing environment may include a distributed computing environment or a stream computing environment. A first portion of the application may be developed by a first user. A second portion of the application may be received by the first user. The second portion of the application may be compiled and developed by a second user. Based on the set of warning data, a curative action for utilization to develop the application may be formulated. The curative action may be carried-out to develop the application.

Warning data management for distributed application development

A method, computer program product, and computer system for consolidating, at a computing device, a number of parallel streams, entering a hardware accelerator in a cluster, into fewer streams. The fewer streams, exiting the hardware accelerator in the cluster, may be returned back into the number of parallel streams.

System and method for hardware acceleration for operator parallelization with streams

A database which receives input from a data streaming application may include in-flight streamed data within the scope of a command which alters database data. Preferably, the streaming application produces data tuples for input to at least one table of the database. A user submitting modify data commands to apply to multiple tuples meeting some specified logical parameters may define the command scope to include in-flight data in the streaming application, causing the database manager to forward the command to the streaming application. A streaming application manager invokes database agents in respective nodes of the streaming application, which identify in-flight data meeting the specified logical parameters of the command, and apply the modify data operation to such data.

Altering in-flight streamed data from a relational database

A tuple testing and routing operator in a streaming application routes data tuples to multiple parallel test operators that test in parallel the data tuples, receives feedback from the multiple parallel test operators regarding the results of testing the data tuples, routes a data tuple to a first operator when the data tuple passes the multiple parallel test operators according to a specified pass threshold, and optionally routes the data tuple to a second operator when the data tuple does not pass the multiple parallel test operators according to the specified pass threshold. The pass threshold allows testing to be done in a way that does not require all tests to be performed for all data tuples, thereby enhancing performance.

Data tuple testing and routing for a streaming application

A method, computer program product, and computing device for identifying at least one tuple of interest from a plurality of tuples during execution of a streaming application. A window of tuples including the at least one tuple from the plurality of tuples may be defined. One or more operations may be performed on the window of tuples.

System and method for defining windows of tuples of interest

Aspects of the disclosure relate to managing a set of warning data with respect to an execution phase in a computing environment. In embodiments, the computing environment may include a distributed computing environment or a stream computing environment. The set of warning data may be detected with respect to the execution phase. In embodiments, the set of warning data may be coupled with a computing artifact. In embodiments, the computing artifact may include a compilation which has a computing object in association with the set of warning data. Using the set of warning data, an execution action which pertains to the computing artifact may be determined. In embodiments, the execution action may include a code deployment to a set of computing units, a run-time check modification, or a process attribute modification. The execution action which pertains to the computing artifact may be performed.

Warning data management with respect to an execution phase

In preventing malicious operator placement in a streaming application, a stream computing management system receives a submission of an operator graph for the streaming application. A scheduler of the stream computing management system places the operators across a plurality of computing nodes. A threat detector of the stream computing management system monitors placements of the operators during the submission of the operator graph and a runtime of the streaming application. The threat detector further monitors runtime activities of the operators of the streaming application. The threat detector compares the runtime activities of the operators of the streaming application with stored activities patterns. The threat detector determines whether any anomalies identified based on the comparing exceeds a risk tolerance threshold. In response to an anomaly exceeding the risk tolerance threshold, the scheduler changes a placement of at least one of the operators.

System for preventing malicious operator placement in streaming applications

A first set of code origination data which corresponds to a first computing object and a second set of code origination data which corresponds to a second computing object may be detected for utilization to develop a streaming application in the stream computing environment. Based on the first and second sets of code origination data, a code assembly arrangement with respect to the first and second computing objects may be determined. Based on the first and second sets of code origination data, a consistent region may be determined to utilize for the code assembly arrangement. The code assembly arrangement may be established, in the consistent region, with respect to the first and second computing objects to develop the streaming application.

Growing community of inventors

Rochester, MN, United States of America

Alexander Cook