International Business Machines Corporation

Christopher R Sabotta

Manuel Orozco

John Matthew Santosuosso

Alexander Cook

David M Koster

Joseph C Schmidt

Alexander J Pogue

Jason Allan Nikolai

Cory J Kleinheksel

A deployment manager deploys processing elements of a streaming application in a non-uniform memory access (NUMA) aware manner to reduce memory coherency overhead in a streaming application. The deployment manager is able to utilize information about an application's operators and the architecture of the NUMA nodes to place whole processing elements on a single NUMA node. Where the operators of a processing element would cross NUMA node boundaries, the deployment manager may consolidate the threads of a processing element to place an application's operators on a single NUMA node to increase efficiency of the system.

Deployment of processing elements in non-uniform memory access environments

A method, computer program product, and computer system for defining, on at least one processor, one or more potential overflow paths for processing at least a portion of a data load through a plurality of stream operators deployed on a plurality of computing devices. Embodiments may include determining, on the at least one processor, a conditional availability for each computing device of the plurality of computing devices. Embodiments may also include identifying, on the at least one processor, a change in one or more of a configuration of the plurality of computing devices and the at least a portion of the data load. Embodiments may further include dynamically activating, on the at least one processor, one or more potential overflow paths in response to the identified change based upon, at least in part, the conditional availability for each computing device of the plurality of computing devices.

System and method for dynamic activation of real-time streaming data overflow paths

A first set of code origination data which corresponds to a first computing object and a second set of code origination data which corresponds to a second computing object may be detected for utilization to develop a streaming application in the stream computing environment. Based on the first and second sets of code origination data, a code assembly arrangement with respect to the first and second computing objects may be determined. Based on the first and second sets of code origination data, a consistent region may be determined to utilize for the code assembly arrangement. The code assembly arrangement may be established, in the consistent region, with respect to the first and second computing objects to develop the streaming application.

Code origination data management for code assembly

A variable checkpoint mechanism in a streams manager checkpoints a streaming application based on periodic time periods for checkpoints. The variable checkpoint mechanism can take a checkpoint before a periodic time period ends when a spike is coming, or can take a checkpoint after the periodic time period ends when there is backpressure in a consistent region of the streaming application. When there is no anticipated spike coming and when there is no backpressure in a consistent region of the streaming application, the checkpoint is performed at the normal end of the periodic time period for checkpoints. In this manner the checkpoint timing of the variable checkpoint mechanism can be adjusted real-time to minimize the negative impact of checkpointing on the performance of the streaming application.

Variable checkpointing in a streaming application with one or more consistent regions

Disclosed aspects relate to managing a set of development data for a stream computing environment. A set of development data related to a computing object may be detected. The set of development data may be derived from application development for utilization in the stream computing environment. The set of development data may be established in association with the computing object. A computing artifact which has the computing object in association with the set of development data may be compiled.

Development data management for a stream computing environment

A streaming application processes data by processing tuples via operators. Bottleneck operators are identified. A degree of streaming speed is maintained, even when non-bottleneck operators participate in increased resiliency operations.

Increasing data resiliency operations based on identifying bottleneck operators

Aspects of the disclosure relate to managing code origination data for a distributed computing environment having a set of compute nodes which includes a first compute node. A first set of code origination data which corresponds to a first computing object may be detected for utilization to develop a distributed application in the distributed computing environment. Using the first set of code origination data, the first compute node may be identified to host the first computing object for the distributed application. The first computing object may be deployed to the first compute node to develop the distributed application.

Code origination data management for host placement

Disclosed aspects relate to checkpointing a set of stream computing data with respect to a stream computing environment having a set of windowed stream operators including both a first windowed stream operator and a second windowed stream operator. It may be identified that the first windowed stream operator has a first subset of the set of stream computing data. It may be identified that the second windowed stream operator has the first subset of the set of stream computing data. It may be determined to checkpoint the first subset of the set of stream computing data without a redundant checkpoint related to the first and second windowed stream operators. The set of stream computing data may be checkpointed without the redundant checkpoint of the first subset of the set of stream computing data.

Checkpointing a set of stream computing data

Systems, methods, and computer program products to perform an operation comprising locking, by a first one of a database management system (DBMS) and a distributed application, one of a first database element and a first data tuple, wherein the first data tuple is of a plurality of tuples in an operator graph of the distributed application, transmitting an indication of the locking by the first one of the DBMS and the streams computing application to a second one of the DBMS and the distributed application, identifying, by the second one of the DBMS and the distributed application, one of a second database element and a second data tuple corresponding to the one of the first database element and the first data tuple, and locking, by the second one of the DBMS and the streams computing application, the identified one of the second database element and the second data tuple.

Cooperative locking between databases integrated with distributed programming environments

Aspects of present disclosure include a computer implemented method, system, and computer program product for implementing an encryption policy in a stream computing environment. The method includes: monitoring a plurality of stream operators within a stream computing environment to collect performance data, identifying a point of congestion within the stream computing environment, based on one or more values within the collected performance data satisfying a predefined congestion condition, determining one or more stream operators of the plurality of stream operators associated with the identified point of congestion, redeploying the one or more stream operators to an isolated network within the computing environment, and modifying an encryption policy for the stream computing environment to disable one or more encryption operations on the redeployed one or more stream operators.

Encryption prioritization for distributed streaming applications

Disclosed aspects relate to tuple traffic management in a stream computing environment to process a stream of tuples using a set of control group data with respect to a set of operation-system-level virtualization and resource isolation containers. The set of control group data may be collected with respect to a set of containers in the stream computing environment. A tuple flow model may be determined with respect to the set of containers in the stream computing environment based on the set of control group data. The stream of tuples may be processed using the tuple flow model with respect to the set of containers in the stream computing environment.

Tuple traffic management

A variable checkpoint mechanism in a streams manager checkpoints a streaming application based on periodic time periods for checkpoints. The variable checkpoint mechanism can take a checkpoint early before a periodic time period ends or late after the periodic time period ends based on predicted size of one or more tuple windows in the streaming application. The time for taking the checkpoint can be selected based on multiple checkpoint timing criteria, which include storage requirement for the checkpoint and predicted backpressure in the flow graph. In this manner the checkpoint timing of the variable checkpoint mechanism can be adjusted real-time to minimize the negative impact of checkpointing on the performance of the streaming application.

Growing community of inventors

Rochester, MN, United States of America

Christopher R Sabotta