International Business Machines Corporation

David M Koster

Jason Allan Nikolai

Alexander Cook

John Matthew Santosuosso

Andrew Thomas Thorstensen

Cory J Kleinheksel

Adam D Reznechek

Christopher R Sabotta

Alexander J Pogue

Manuel Orozco

Jonathan Louis Kaus

Joseph C Schmidt

Michael J Branson

Peter Andrew Nicholls

Schuman M Shao

Aditi Rajagopal

Warren Patrick Acker

Steven Edward Royer

Daniel Mark Hursh

A method comprises establishing a packet size limit. The packet size limit may govern the maximum permitted size of packets that are transmitted between a first node and a second node of a network. The method also comprises monitoring a buffer of the second node in the network. The method also comprises determining, based on the monitoring, that the buffer of the second node is filled above an upper capacity threshold. The method also comprises increasing, based on the determining, the first packet size limit.

Real-time adjustment of packet size limit in virtual networks

An XML fragmenting mechanism uses an XML schema for the XML file to split up the XML file in a hierarchal structure of data blocks for storage in a storage system with a limited block size such as a cluster coordination service. The XML fragmenting mechanism creates an XML file map to document the structure of the XML file in the storage system. The XML fragmenting mechanism stores the data blocks in the storage system according to the XML file map and supports retrieval of all or part of the data in a format that supports XML validation.

Intelligent XML file fragmentation

Techniques for optimizing updates to a distributed data processing application are provided. A runtime engine compares a first version of a distributed application to a second, new version of the distributed application. The runtime engine identifies change(s) to the first version. The runtime engine generates compiler hint(s) for the change(s). The second version of the distributed application and compiler hint(s) are sent to a compiler. The runtime engine receives, from the compiler, a compiled second version of the distributed application and compiler-scheduler hint(s) describing respective optimizations the compiler made to the compiled second version of the distributed application based on the change(s) made to the first version of the distributed application and the compiler hint(s). The runtime engine optimizes an update by overlaying the change(s) made to the first version of the distributed application based on the compiler-scheduler hint(s) and the compiled second version of the distributed application.

Optimizing updates to distributed data processing applications

A computer-implemented method, in an example, may include calculating a time-to-live value for at least one packet based on a hop count between each of a plurality of processing element containers within an application streaming network; updating at least one processing element container output connection based on the calculated time-to-live value; and monitoring a streams resource metrics service for a change in a packet delivery rate.

Streaming network

A connection can be made to a processing element of a remotely deployed and live streaming application executed by a first data processing system, the processing element containing at least one operator that processes at least one tuple. As the live streaming application is executed, without slowing or modifying data flow of the live streaming application execution to client devices, a copy of the tuple and a memory dump of state data for a state of the operator can be received, and the tuple can be tracked through a call graph. The state data can be loaded into a local instance of the operator loaded into a debugger. At least a portion of the call graph can be presented to a user, and a flow of the tuple through the call graph based on the state data for the operator can be indicated.

Debugging a live streaming application

In using a virtual extensible local area network (VXLAN) for stream processing, a management system allocates resources for a streaming application based on an operator graph of the streaming application. The management system assigns the resources to a group of VXLAN segments based on the operator graph of the streaming application. A first processing element of the streaming application multicasts data on the group of VXLAN segments. A second processing element on a given VXLAN segment of the group of VXLAN segments receives the data. If the second processing element is an intended recipient of the data, then the second processing element processes the data. If the second processing element is not the intended recipient of the data, then the second processing element ignores the data.

Stream processing without central transportation planning

A method, computer program product, and computer system for identifying, in a dynamically formed cloud, a plurality of voluntary compute nodes, wherein the plurality of voluntary compute nodes may include personal computing devices of a plurality of users, and wherein at least a portion of the plurality of voluntary compute nodes may be configured to communicate with each other to receive and send streaming data. A first compute node of at least the portion of the plurality of voluntary compute nodes may be identified to receive and distribute a streams operator of a streams graph requested by a client computing device. A second compute node of at least the portion of the plurality of voluntary compute nodes may be identified for verifying computational correctness of a tuple stream computation for the stream operator by replicating the tuple stream computations for the stream operator on the second compute node. A final result tuples of the graph may be sent to the client computing device.

System and method for dynamic cloud formations for stream computing

Provided are techniques for operator isolation based on data security requirements. At a cloud node, a graph is received that includes ingest portions of data and operators. For each of the operators, it is determined whether the operator processes protected data. In response to determining that the operator is tagged with an indication that the operator processes protected data, the operator is forwarded to a tenant secure node for processing. In response to determining that the operator is not tagged with an indication that the operator processes protected data, the operator is forwarded to a tenant general node for processing. Then, while the tenant general node is processing the operator, in response to determining that the operator is processing protected data a tag is associated with the operator to indicate that the operator processes protected data and the operator is forwarded to the tenant secure node for processing.

Operator isolation based on data security requirements

System resource usage by a streaming application processing workloads can be monitored, the streaming application comprising at least one parallelizable operator, at least a first instance of the parallelizable operator being initialized to execute. Based on the monitoring, at least one performance metric for the streaming application in processing the workloads can be determined. Based on the at least one performance metric, whether hosting of the streaming application is, or is near to, violating at least one criteria can be determined. If so, at least one additional instance of at least one parallelizable operator of the streaming application that is currently executing can be initialized. Responsive to initializing the at least one additional instance of at least one of the parallelizable operators, a portion of the workloads can be directed to the at least one additional instance of at least one of the parallelizable operators.

Optimizing streaming graph topology based on service level agreement

Disclosed aspects relate to cache management in a stream computing environment that uses a set of many-core hardware processors to process a stream of tuples by a plurality of processing elements which operate on the set of many-core hardware processors. The stream of tuples to be processed by the plurality of processing elements which operate on the set of many-core hardware processors may be received. A tuple-processing hardware-route on the set of many-core hardware processors may be determined based on a cache factor associated with the set of many-core hardware processors. The stream of tuples may be routed based on the tuple-processing hardware-route on the set of many-core hardware processors. The stream of tuples may be processed by the plurality of processing elements which operate on the set of many-core hardware processors.

Cache management in a stream computing environment that uses a set of many-core hardware processors

A computer system configures processing elements within a distributed computing system. A processing element within a distributed computing environment is determined to be affected by a software update, wherein each processing element of the distributed computing system includes a plurality of components and the software update modifies the components of the processing elements. The determined processing element is split into a plurality of processing elements based on a set of factors. The plurality of components of the processing element are assigned among the plurality of processing elements based on components affected by the software update. Embodiments of the present invention further include a method and program product for configuring processing elements within a distributed computing system in substantially the same manner described above.

Fusing and unfusing operators during a job overlay

A method, computer program product, and computer system for consolidating, at a computing device, a number of parallel streams, entering a hardware accelerator in a cluster, into fewer streams. The fewer streams, exiting the hardware accelerator in the cluster, may be returned back into the number of parallel streams.

System and method for hardware acceleration for operator parallelization with streams

A deployment manager deploys processing elements of a streaming application in a non-uniform memory access (NUMA) aware manner to reduce memory coherency overhead in a streaming application. The deployment manager is able to utilize information about an application's operators and the architecture of the NUMA nodes to place whole processing elements on a single NUMA node. Where the operators of a processing element would cross NUMA node boundaries, the deployment manager may consolidate the threads of a processing element to place an application's operators on a single NUMA node to increase efficiency of the system.

Deployment of processing elements in non-uniform memory access environments

A method, computer program product, and computer system for defining, on at least one processor, one or more potential overflow paths for processing at least a portion of a data load through a plurality of stream operators deployed on a plurality of computing devices. Embodiments may include determining, on the at least one processor, a conditional availability for each computing device of the plurality of computing devices. Embodiments may also include identifying, on the at least one processor, a change in one or more of a configuration of the plurality of computing devices and the at least a portion of the data load. Embodiments may further include dynamically activating, on the at least one processor, one or more potential overflow paths in response to the identified change based upon, at least in part, the conditional availability for each computing device of the plurality of computing devices.

Growing community of inventors

Rochester, MN, United States of America

David M Koster