International Business Machines Corporation

Microsoft Technology Licensing, LLC

Henry Chiu

Bradley William Fawcett

Jason Allan Nikolai

Jingdong Sun

Paul Gerver

Tse-Ching Yu

Computer software that causes a stream processing management node to perform the following operations: (i) establishing data communication between the stream processing management node and a plurality of stream processing instances executing on respective computing environments in a multi-environment computing system; (ii) distributing one or more processing units of a stream processing job to a first set of stream processing instances of the plurality of stream processing instances; (iii) receiving, from the one or more stream processing instances of the first set of stream processing instances, processing results associated with the one or more processing units of the stream processing job; and (iv) performing a machine learning based stream management operation based, at least in part, on the received processing results.

Selection of stream management operations based on machine learning in a distributed computing environment

Computer software executing on computer hardware that performs the following operations: (i) training a machine learning model to determine allocations of computing resources to processing elements of a stream processing job according to a specified objective; and (ii) allocating a set of computing resources to the processing elements by: allocating to the processing elements a first subset of the set of computing resources based, at least in part, on a minimum resource requirement for the processing elements, and allocating to the processing elements a second subset of the set of computing resources based, at least in part, on an allocation determined using the trained machine learning model.

Optimizing resource allocation for distributed stream processing systems

A method for maintaining version consistency of resources. The method provides for one or more processors to receive a submitted request to run a job in which the job includes a processing element and a timestamp associated with running the job. Identification of a resource type associated with the processing element is determined, based on a tag included in the job, associated with the processing element. A version of the resource type of the processing element is determined, based on a mapping of the tag associated with the identified resource type and the timestamp of the job. The resource type of the determined version is requested from a resource manager, and responsive to a confirmation of assigning the version of the resource type from the resource manager, the process element of the job is performed on the version of the resource type assigned by the resource manager.

Maintaining stream processing resource type versions in stream processing

In response to receiving a flow of data, one or more operators in a parallel region included in a distributed data processing system for stream processing the flow of data are initialized. In response to identifying a change to the flow of data or a change to the distributed data processing system, at least one of the flow of data and a size of the parallel region included in the distributed data processing system are dynamically adjusted. The flow of data is processed via the dynamically adjusted one of the flow of data and the parallel region included in the distributed data processing system.

Load balancing in streams parallel regions

A stream of tuples to be processed by processing elements that operate on one or more compute nodes of a stream application is monitored. Each processing element has one or more stream operators. A program request to execute a set of program code instructions is received. A stream application target of the set of program code instructions is identified based on the program request. A portion of the stream application is encapsulated with a generic program execution structure during operation of the stream application. The structure is configured to receive and execute program code instructions outside of a preconfigured operation of the stream application. The set of program code instructions is executed by the structure during operation of the stream application, at a time period of execution of the portion, in response to the program request, and based on the stream application target.

Aspect aware streams computing

A stream of tuples is monitored. The stream of tuples is to be processed by a plurality of processing elements of a stream application that operate on one or more compute nodes, each processing element having one or more stream operators. A processing rate of a first stream operator of the stream application is calculated. The processing rate is based on the number of tuples that are processed by the first stream operator. It is determined that the processing rate of the first stream operator meets a predetermined tuple processing criterion. The processing rate of the first stream operator is adjusted based on the predetermined tuple processing criterion.

Streams runtime processing rate adjustment

A method, system and computer program product for facilitating remote debugging of parallel regions in stream computing applications. A stream computing management server (SCMS) communicates a list of processing elements to a debugging interface. Responsive to setting a debugging breakpoint for a processing element of the list of processing elements, the SCMS receives a command to enable remote debugging for the selected processing element. In this regard, the processing element is a part of a parallel channel in a distributed processing environment. The SCMS maps the processing element to an attachment information in the distributed environment. The SCMS dynamically attaches a remote debugger to the processing element based on the attachment information.

Remote debugging parallel regions in stream computing applications

Within a portion of a currently executing compiled streams application according to a request to modify a configuration of the portion, a placement is performed of an operator of the currently executing compiled streams application into a processing element, wherein the currently executing compiled streams application has a configuration assigned before application execution began. In response to the placement, a resource allocation corresponding to the placement is performed according to the request, the resource allocation specifying a resource assigned to the processing element. The processing element is activated to execute the operator according to the placement and the resource allocation.

Operator to processing element assignment in an active stream processing job

An optical disc drive (ODD) includes one or more sensors which detect a user placing a disc in the loading path. In response, a process is triggered for sensing the thickness of a protective layer of the disc even while the disc is moving toward a turntable. An initial determination of the type of the disc (e.g. CD, DVD or Blu-ray&#xae;) can be determined based on the thickness and reported to a host computing device. The host, in turn, determines an application to load based on the disc type. The loading of the application can begin sooner than if the ODD waited until the disc was seated on the turntable to detect the disc type. A final determination of the disc type can be made while the disc is seated as well. The host is informed if there is a disagreement between the initial determination and the final determination.

Growing community of inventors

San Jose, CA, United States of America

Henry Chiu