International Business Machines Corporation

Eric L Barsness

John Matthew Santosuosso

Michael J Branson

David Lee Darrington

Amanda Peters

Daniel Beuch

Alexander Cook

James E Carey

Joseph W Cropper

Jay Symmes Bryant

Richard D Dettinger

Randy William Ruhlow

Amanda Randles

Jeremy Alan Arnold

Mahdad Majd

Brian Robert Muras

Paul S Nelsestuen

Manuel Orozco

Brian Edward Olson

Cary Lee Bates

Jason Allan Nikolai

Ryan Kirk Cradick

Robert J Bestgen

Michael Brian Brutman

Bradford Lee Cobb

Michael D Pfeifer

Patrick Joseph McCarthy

Peter John Johnson

Michael P Koranda

Richard Fred Harwood

Ray LeRoy Lucas

Jennifer Anne Larson

John Joseph Stecher

Mark Robert Funk

Ross Evan Johnson

Theresa Renee Euler

Douglas David Jans

John A Stroh

John M Santousso

Daniel R Hill

Feedback tuples are implemented in a stream computing environment to provide data flow status information for use in detecting, identifying and/or handling data flow anomalies for a stream computing application.

Feedback tuples for detecting data flow anomalies in stream computing environment

Queries are monitored in a database which receives input from a stream computing application to identify data of interest. Parameters defining the data of interest, which are preferably expressed as a logical query, are sent to the stream computing application, which then processes the in-flight streamed data satisfying the parameters in some special manner. In some embodiments, the stream computing application increases the processing priority of in-flight data satisfying the parameters. In some embodiments, the stream computing application applies additional processing steps to the in-flight data satisfying the parameters to provide enhanced data or metadata.

Managing the processing of streamed data in a data streaming application using query information from a relational database

Certain aspects of the present disclosure provide techniques for identifying temporal data in data streams to create a temporal database for a stream(s) application to query for temporal data. An example technique includes receiving streams of data at a streams engine and processing the streams of data according to a priority order. The streams engine identifies whether the database is a temporal database and identifies temporal data in each stream of data based on frame analysis, natural language processing techniques, metadata, and optical character recognition. Further, the streams engine generates context data corresponding to the temporal data. The streams engine generates a temporal data record based on the temporal data and context data, and the streams engine generates a reliability factor. The temporal data record and reliability factor are stored in the temporal database for a stream application to query regarding temporal information at a later point in time.

Integrating relational database temporal tables with a distributed programming environment

In a streaming application receiving streaming data to be processed by a plurality of processing elements (PEs), one of a plurality of upstream PEs that reside on separate compute nodes and transmit data to a multi-input port PE is selected based on a data flow rate measured between at least two components in the plurality of PEs (e.g., an amount of incoming data per unit time or an amount of network bandwidth). At least one operator from the selected upstream PE is redeployed into the multi-input port PE. In some embodiments, the operator is selected from among a plurality of operators fused together in the selected upstream PE, un-fused from the selected upstream PE, and fused into the multi-input port PE. In some embodiments, a dynamic fusing capability is realized by selecting different ones of the plurality of upstream PEs in response to changes in the measured data flow rate.

Autonomic fusion changes based off data rates

Profiling data characterizing a data streaming application is used to predict data which will need to be retrieved by a processing element during execution of the data streaming application. Data is retrieved responsive to the prediction, in advance of actual demand by the processing element which requires it. Prediction may be based at least in part on upstream tuple contents, and could include other historical data retrieval patterns. In some embodiments, retrieval of predicted data may be delayed so that data is retrieved just in time.

Reducing flow delays in a data streaming application caused by lookup operations

Disclosed aspects include the database management system detecting that a first data storage medium includes a first set of data of a database. The first data storage medium has a first access-factor. The database management system detects that a second data storage medium includes a second set of data of the database. The second data storage medium has a second access-factor which differs from the first access-factor. Based on a data storage relationship, the database management system determines a query plan to access both the first set of data with respect to the first data storage medium and the second set of data with respect to the second data storage medium. The database management system establishes the query plan.

Query plan based on a data storage relationship

A method, system and computer program product for staggering a stream application's deployment, by: providing one or more configuration settings that define one or more delays for instantiation or initialization of at least one target processing element of the stream application based on the stream application's run-time conditions or events; and instantiating or initializing the target processing element of the stream application when the delays defined by the configuration settings have been satisfied. The configuration settings may define the delays to wait until a specified period of time has lapsed after one or more other processing elements have been instantiated or initialized, or to wait until a specified amount of data has been processed by one or more other processing elements, or to wait until one or more signals is received from one or more other processing elements, before instantiating or initializing the target processing element.

Staggering a stream application's deployment

Aspects of the disclosure relate to using a set of warning data to manage distributed development of an application in a computing environment. In embodiments, the computing environment may include a distributed computing environment or a stream computing environment. A first portion of the application may be developed by a first user. A second portion of the application may be received by the first user. The second portion of the application may be compiled and developed by a second user. Based on the set of warning data, a curative action for utilization to develop the application may be formulated. The curative action may be carried-out to develop the application.

Warning data management for distributed application development

Disclosed aspects relate to window management in a stream computing environment. A set of computing resources may be detected with respect to the stream computing environment. Based on the set of computing resources, a set of window configurations in the stream computing environment may be determined. In response to determining the set of window configurations in the stream computing environment, the set of window configurations may be established in the stream computing environment.

Window management based on a set of computing resources in a stream computing environment

A computer system controls processing elements associated with a stream computing application. A stream computing application is monitored for the occurrence of one or more conditions. One or more processing element groups are determined to be restarted based on occurrence of the one or more conditions, wherein the processing element groups each include a plurality of processing elements associated with the stream computing application. Each processing element of the determined one or more processing element groups is concurrently restarted. Embodiments of the present invention further include a method and program product for controlling processing elements within a stream computing application in substantially the same manner described above.

Controlling processing elements in a distributed computing environment

A database which receives input from a data streaming application may include in-flight streamed data within the scope of a command which alters database data. Preferably, the streaming application produces data tuples for input to at least one table of the database. A user submitting modify data commands to apply to multiple tuples meeting some specified logical parameters may define the command scope to include in-flight data in the streaming application, causing the database manager to forward the command to the streaming application. A streaming application manager invokes database agents in respective nodes of the streaming application, which identify in-flight data meeting the specified logical parameters of the command, and apply the modify data operation to such data.

Altering in-flight streamed data from a relational database

A method for processing distributed breakpoints when debugging a distributed application includes establishing a breakpoint in source code of a distributed computer program and executing the distributed computer program. Thereafter, an encountering of the breakpoint is detected in one of the processes, and execution of the one of the processes halted. However, halting execution of any other of the processes is delayed for a delay period during which a list both is generated of others of the processes in which the breakpoint is encountered and also is sorted in accordance with prioritization criteria. Finally, the sorted list is presented after the lapse of the delay period, individual ones of the processes selected in the list, and execution of the selected individual ones of the processes in the sorted list halted while halting of execution of non-selected ones of the processes in the sorted list is bypassed.

Intelligent processing of distributed breakpoints

Growing community of inventors

Pine Island, MN, United States of America

Eric L Barsness