International Business Machines Corporation

Bradley William Fawcett

Jason Allan Nikolai

Henry Chiu

Jingdong Sun

Paul Gerver

Frederick Douglis

Fan Ye

James E Carey

Michael J Branson

Zhen Liu

Timothy James Graser

Brent A Carlson

William Waller

Anton Viktorovich Riabov

Lee Anton Sendelbach

Curtis Howard Brobst

Anders Magnus Nilsson

Tore Magnus Dahl

Vincent Edmund Price

Dave Henry Gross

Bin Rong

Siegfried Wiesenhofer

Bo An

Wilfried Lesnik

Computer software that causes a stream processing management node to perform the following operations: (i) establishing data communication between the stream processing management node and a plurality of stream processing instances executing on respective computing environments in a multi-environment computing system; (ii) distributing one or more processing units of a stream processing job to a first set of stream processing instances of the plurality of stream processing instances; (iii) receiving, from the one or more stream processing instances of the first set of stream processing instances, processing results associated with the one or more processing units of the stream processing job; and (iv) performing a machine learning based stream management operation based, at least in part, on the received processing results.

Selection of stream management operations based on machine learning in a distributed computing environment

Computer software executing on computer hardware that performs the following operations: (i) training a machine learning model to determine allocations of computing resources to processing elements of a stream processing job according to a specified objective; and (ii) allocating a set of computing resources to the processing elements by: allocating to the processing elements a first subset of the set of computing resources based, at least in part, on a minimum resource requirement for the processing elements, and allocating to the processing elements a second subset of the set of computing resources based, at least in part, on an allocation determined using the trained machine learning model.

Optimizing resource allocation for distributed stream processing systems

A method for maintaining version consistency of resources. The method provides for one or more processors to receive a submitted request to run a job in which the job includes a processing element and a timestamp associated with running the job. Identification of a resource type associated with the processing element is determined, based on a tag included in the job, associated with the processing element. A version of the resource type of the processing element is determined, based on a mapping of the tag associated with the identified resource type and the timestamp of the job. The resource type of the determined version is requested from a resource manager, and responsive to a confirmation of assigning the version of the resource type from the resource manager, the process element of the job is performed on the version of the resource type assigned by the resource manager.

Maintaining stream processing resource type versions in stream processing

In response to receiving a flow of data, one or more operators in a parallel region included in a distributed data processing system for stream processing the flow of data are initialized. In response to identifying a change to the flow of data or a change to the distributed data processing system, at least one of the flow of data and a size of the parallel region included in the distributed data processing system are dynamically adjusted. The flow of data is processed via the dynamically adjusted one of the flow of data and the parallel region included in the distributed data processing system.

Load balancing in streams parallel regions

A stream of tuples to be processed by processing elements that operate on one or more compute nodes of a stream application is monitored. Each processing element has one or more stream operators. A program request to execute a set of program code instructions is received. A stream application target of the set of program code instructions is identified based on the program request. A portion of the stream application is encapsulated with a generic program execution structure during operation of the stream application. The structure is configured to receive and execute program code instructions outside of a preconfigured operation of the stream application. The set of program code instructions is executed by the structure during operation of the stream application, at a time period of execution of the portion, in response to the program request, and based on the stream application target.

Aspect aware streams computing

A stream of tuples is monitored. The stream of tuples is to be processed by a plurality of processing elements of a stream application that operate on one or more compute nodes, each processing element having one or more stream operators. A processing rate of a first stream operator of the stream application is calculated. The processing rate is based on the number of tuples that are processed by the first stream operator. It is determined that the processing rate of the first stream operator meets a predetermined tuple processing criterion. The processing rate of the first stream operator is adjusted based on the predetermined tuple processing criterion.

Streams runtime processing rate adjustment

A method, system and computer program product for facilitating remote debugging of parallel regions in stream computing applications. A stream computing management server (SCMS) communicates a list of processing elements to a debugging interface. Responsive to setting a debugging breakpoint for a processing element of the list of processing elements, the SCMS receives a command to enable remote debugging for the selected processing element. In this regard, the processing element is a part of a parallel channel in a distributed processing environment. The SCMS maps the processing element to an attachment information in the distributed environment. The SCMS dynamically attaches a remote debugger to the processing element based on the attachment information.

Remote debugging parallel regions in stream computing applications

Within a portion of a currently executing compiled streams application according to a request to modify a configuration of the portion, a placement is performed of an operator of the currently executing compiled streams application into a processing element, wherein the currently executing compiled streams application has a configuration assigned before application execution began. In response to the placement, a resource allocation corresponding to the placement is performed according to the request, the resource allocation specifying a resource assigned to the processing element. The processing element is activated to execute the operator according to the placement and the resource allocation.

Operator to processing element assignment in an active stream processing job

According to a request to modify a configuration of the portion, a portion of a currently executing compiled streams application is modified by computing a current composite load on a resource used by an executing processing element. As a part of modifying the configuration, within the portion according to the request, a composite load on a first candidate resource is predicted. Based on the predicted composite load, a configuration modification is generated based on a predicted throughput improvement resulting from migrating the processing element to the first candidate resource. The processing element is migrated, resulting in a throughput improvement, to a target resource selected from a set of candidate resources, a resource in the set of candidate resources categorized based on a predicted second composite load resulting from migrating the processing element to the second candidate resource according to the configuration modification.

Multiple metric based load prediction and resource allocation in an active stream processing job

Disclosed aspects relate to configuration management in a stream computing environment to process a stream of tuples using a compiled application bundle. A set of configuration overlay parameters may be established separate from the compiled application bundle. A set of configuration overlay parameter values may be ascertained with respect to the set of configuration overlay data. A stream environment application overlay configuration may be determined based on the set of configuration overlay parameter values. The stream of tuples may be processed using the stream environment application overlay configuration.

Configuration management in a stream computing environment

A method, computer system, and a computer program product for managing asset placement with respect to a distributed computing environment having a set of hosts is provided. The present invention may include detecting a set of host computing resource requirement data for an asset, wherein the distributed computing environment includes a stream computing environment and the asset includes a rookie asset in the stream computing environment. The present invention may include identifying a set of computing resource profile data for a set of hosts in the distributed computing environment. The present invention may include determining, by comparing the set of host computing resource requirement data for the asset and the set of computing resource profile data for the set of hosts, an asset placement arrangement. The present invention may include establishing, based on the asset placement arrangement, the asset in the distributed computing environment.

Managing asset placement with respect to a distributed computing environment having a set of hosts

Disclosed aspects relate to operator fusion management in a stream computing environment. A topology model which indicates a set of stream operators, a set of connections between the set of stream operators, and a set of stream operator attributes for the set of stream operators may be established. Based on the topology model, a set of operator fusion management operations to combine the set of stream operators into a set of processing elements may be determined. The set of processing elements may be constructed by performing the set of operator fusion management operations.

Operator fusion management in a stream computing environment

Disclosed aspects relate to managing a set of compute nodes for processing a stream of tuples using a set of processing elements. The set of compute nodes is structured to include both a first compute node having a first configuration and a second compute node having a second configuration. The first configuration differs from the second configuration. Based on the first configuration and the set of processing elements which includes a first processing element, a determination is made to establish the first processing element on the first compute node and the first processing element is established on the first compute node. In embodiments, based on the second configuration and the set of processing elements which includes a second processing element, a determination is made to establish the second processing element on the second compute node and the second processing element is established on the second compute node.

Growing community of inventors

Byron, MN, United States of America

Bradley William Fawcett