International Business Machines Corporation

Jingdong Sun

Jessica R Eidem

Brian Robert Muras

Bin Cao

Rafal Przemyslaw Konik

Roger Alan Mittelstadt

Jason Allan Nikolai

Bradley William Fawcett

Henry Chiu

Paul Gerver

Jian Tang

Xiaoming Yu

Jingli Zhang

Qian Sun

Computer software that causes a stream processing management node to perform the following operations: (i) establishing data communication between the stream processing management node and a plurality of stream processing instances executing on respective computing environments in a multi-environment computing system; (ii) distributing one or more processing units of a stream processing job to a first set of stream processing instances of the plurality of stream processing instances; (iii) receiving, from the one or more stream processing instances of the first set of stream processing instances, processing results associated with the one or more processing units of the stream processing job; and (iv) performing a machine learning based stream management operation based, at least in part, on the received processing results.

Selection of stream management operations based on machine learning in a distributed computing environment

A sharable resource of a first user's environment is identified. The sharable resource is configured as sharable in a shared computer environment. A matching resource that is sufficiently similar to the sharable resource is located. The matching resource is used by pre-existing users of the shared computer environment. Agreement from the pre-existing users for the first user to access the matching resource is obtained. The first user is then provided access to the matching resource.

Flexible sharing in shared computer environments

Computer software executing on computer hardware that performs the following operations: (i) training a machine learning model to determine allocations of computing resources to processing elements of a stream processing job according to a specified objective; and (ii) allocating a set of computing resources to the processing elements by: allocating to the processing elements a first subset of the set of computing resources based, at least in part, on a minimum resource requirement for the processing elements, and allocating to the processing elements a second subset of the set of computing resources based, at least in part, on an allocation determined using the trained machine learning model.

Optimizing resource allocation for distributed stream processing systems

A method for maintaining version consistency of resources. The method provides for one or more processors to receive a submitted request to run a job in which the job includes a processing element and a timestamp associated with running the job. Identification of a resource type associated with the processing element is determined, based on a tag included in the job, associated with the processing element. A version of the resource type of the processing element is determined, based on a mapping of the tag associated with the identified resource type and the timestamp of the job. The resource type of the determined version is requested from a resource manager, and responsive to a confirmation of assigning the version of the resource type from the resource manager, the process element of the job is performed on the version of the resource type assigned by the resource manager.

Maintaining stream processing resource type versions in stream processing

In response to receiving a flow of data, one or more operators in a parallel region included in a distributed data processing system for stream processing the flow of data are initialized. In response to identifying a change to the flow of data or a change to the distributed data processing system, at least one of the flow of data and a size of the parallel region included in the distributed data processing system are dynamically adjusted. The flow of data is processed via the dynamically adjusted one of the flow of data and the parallel region included in the distributed data processing system.

Load balancing in streams parallel regions

A stream of tuples to be processed by processing elements that operate on one or more compute nodes of a stream application is monitored. Each processing element has one or more stream operators. A program request to execute a set of program code instructions is received. A stream application target of the set of program code instructions is identified based on the program request. A portion of the stream application is encapsulated with a generic program execution structure during operation of the stream application. The structure is configured to receive and execute program code instructions outside of a preconfigured operation of the stream application. The set of program code instructions is executed by the structure during operation of the stream application, at a time period of execution of the portion, in response to the program request, and based on the stream application target.

Aspect aware streams computing

In an embodiment, a method for controlling a database in a streaming application includes receiving, by a compute node, data for transmission as streaming data on a network connected to the compute node and a database. The compute node then assembles a body portion of a tuple to include the data and assembles a header portion of the tuple to include a database statement that identifies a change to be made by the database to an aspect of the database in a format that is identifiable by the database. The compute node then transmits the tuple having the header portion thereof and the body portion thereof on the network as streaming data with the header and body portions of the tuple being directed to the database.

Streaming tuple database control

A stream of tuples is monitored. The stream of tuples is to be processed by a plurality of processing elements of a stream application that operate on one or more compute nodes, each processing element having one or more stream operators. A processing rate of a first stream operator of the stream application is calculated. The processing rate is based on the number of tuples that are processed by the first stream operator. It is determined that the processing rate of the first stream operator meets a predetermined tuple processing criterion. The processing rate of the first stream operator is adjusted based on the predetermined tuple processing criterion.

Streams runtime processing rate adjustment

A method, system and computer program product for facilitating remote debugging of parallel regions in stream computing applications. A stream computing management server (SCMS) communicates a list of processing elements to a debugging interface. Responsive to setting a debugging breakpoint for a processing element of the list of processing elements, the SCMS receives a command to enable remote debugging for the selected processing element. In this regard, the processing element is a part of a parallel channel in a distributed processing environment. The SCMS maps the processing element to an attachment information in the distributed environment. The SCMS dynamically attaches a remote debugger to the processing element based on the attachment information.

Remote debugging parallel regions in stream computing applications

A tuple manager of a database system processes partial tuples from a streaming application and stores them in a database. The partial tuples may include a large object (LOB) that arrives at the database at a different time than the rest of the corresponding tuple. A tuple manager stores partial tuples and uses a partial tuples index to track the partial tuples and coordinate recombination of corresponding partial tuples. The database allows queries to be run on the partial data before the tuples are reconstructed allowing faster access to potentially important data before the arrival and processing of a partial tuple such as an LOB.

Storing partial tuples from a streaming application in a database system

A streams manager includes a connection converter that allows converting at run-time a static connection to a dynamic connection, and converting at run-time a dynamic connection to a static connection. One or more conversion criteria are defined that determine when a conversion of one or more connections is needed at run-time. When conversion criteria for converting a static connection from a first processing element to a second processing element to a dynamic connection is satisfied, the first processing element interacts with the streams manager and with the second processing element to dynamically change the static connection to a dynamic connection. When conversion criteria for converting a dynamic connection from a first processing element to a second processing element to a static connection is satisfied, the first processing element interacts with the streams manager and with the second processing element to dynamically change the dynamic connection to a static connection.

Dynamically converting static and dynamic connections in a streaming application

Embodiments of the preset disclosure include a method, a system, and a computer program product for rearranging a set of filters within a data streams environment. The method includes setting a first filter ordering of two or more filter processing elements. The method includes determining a filtering efficiency of the first filter ordering. The determining of the filtering efficiency includes monitoring a runtime, one or more tuples, and an amount of tuples retained after filtering of the first filter ordering. The method includes creating a second filter ordering of the two or more filter processing elements. The method includes comparing the filtering efficiency of the second filter ordering to the first filter ordering. The method includes selecting the second filter ordering in response to comparing the filtering efficiency of the second filter ordering to the filtering efficiency of the first filter ordering.

Filter optimizer for data streams

Dynamic connections between processing elements of a streaming application are managed in a distributed manner by the processing elements to increase efficiency and performance of real-time connection updates. A streams application management service includes a connection manager that notifies all processing elements when a job with a new processing element is submitted. Each processing element dynamically maintains its own connections based on one or more of the following conditions: connection data received from the streams management service, and a behavior change of the processing element. Then the processing element reports connection updates to the connection manager in the streams management service. The connection manager may store updated connection information in the connection data.

Distributed management of dynamic processing element connections in streaming applications

A public operator processes data streams from multiple operators in different streaming applications to reduce resource costs and increase efficiency in a streaming system. The public operator uses tuple level security with a unique key for each streaming application to securely process the data streams. A stream security module (SSM) manages encryption to and from the public operators to insure other streaming applications with access to the shared public operator don't have access to data of other applications that may belong to other customers or users. The stream security module may be incorporated into the streams manager of a streaming system.

Tuple level security for streams processing

Method, system, and computer program product are provided to guarantee no tuple loss and/or corruption in a real-time data streaming environment. In some embodiments, a stream of tuples is sent from an entry operator to downstream operator(s), and, at the entry operator, a mark is placed on each tuple in the stream of tuples. In an embodiment where the stream of tuples includes first, second, and subsequent tuple(s), for example, the mark may comprise a tuple-unique number and a tuple-data hash of data contained in the tuple, and the mark on the second and each subsequent tuple may further comprise a link-back hash associated with the immediately preceding tuple in the stream of tuples. In some embodiments, the link-back hash is used to check whether a tuple is lost or received out of order, and the tuple-data hash is used to check whether a received tuple is corrupt.

Growing community of inventors

Rochester, MN, United States of America

Jingdong Sun