International Business Machines Corporation

Kun-Lung Wu

Bugra Gedik

Philip Shi-lung Yu

Henrique Andrade

Joel Leonard Wolf

Rohit Madhukar Khandekar

Vibhore Kumar

Shyh-Kwei Chen

Martin J Hirzel

Philip Shilung Yu

Deepak Rajan

Sujay Sunil Parekh

Gabriela Jacques Da Silva

Kirsten Weale Hildrum

Scott A Schneider

Robert J Soule

Hua Wang

Joseph L Hellerstein

Alexander M Keller

Ming-Syan Chen

Gee-Gwo Mei

Vijaya Krishnan

Xiaolan Zhang

Michail Vlachos

Rohit Wagle

Gang Luo

Ahmed Eldawy

Xiang Ni

Qiong Zou

Fang Zheng

Giuliano Losa

Mark Gee-Gwo Mei

Chung-Sheng Li

Charu Chandra Aggarwal

John Richard Smith

Sriram Kolinjivadi Padmanabhan

Yuan-Chi Chang

Ching-Yung Lin

Asit Dan

Milind R Naphade

Min Wang

Srinivasan Parthasarathy

Rui Hou

Yinglong Xia

Jimeng Sun

Matthew Leon Hill

Andrew Lawrence Frenkiel

Yun-Wu Huang

Huayong Wang

Murray Scott Campbell

Philip Shi-Lung Yu

Ming-Ling Lo

Mark Peter Mendell

Leman Akoglu

Bhuvan Bamba

Robert Grimm

Harish Ragavan

James P Crosskey

Mindi Yuan

Jim Challenger

Howard W Nasgaard

Sunjay S Parekh

Rohit M Kandekar

Spyridon Papadimitrion

Mark (Gee-Gwo) Mei

Philip S-l Yu

Kevin Michael Jordan

Yuzhe Tang

Philip Sl Yu

Embodiments for automating multidimensional elasticity for streaming applications in a computing environment. Each operator in a streaming application may be identified and assigned into one of a variety of groups according to similar performance metrics. One or more threading models may be adjusted for one or more of the groups to one or more different regions of the streaming application.

Automated multidimensional elasticity for streaming application runtimes

In an approach to improve online path profiling by identifying heavy trafficked paths in streaming applications embodiments track data tuples through a streaming application. Embodiments consist of periodically executing a tuple submission application programming interface (API) and a tuple processing API. Additionally, embodiments record active paths using a profiler thread by taking snapshots of a current path index across threads. Furthermore, embodiments generate path usage information based on the recorded active path, generate a path usage graph based on the path usage information; and optimize identified trafficked paths based on the path usage data and path usage map.

Path profiling for streaming applications

A system, method and computer program product for adaptive incremental checkpointing an operator state in a streaming application. The system and method enable reduced costs of checkpointing an operator state in a streaming application, by i) logging updates to operator state and checkpointing operator state in either base (full state) or delta (logged updates) form, ii) dynamically and adaptively adjusting checkpointing options, and iii) maintaining dependencies between checkpoints to allow automatic restoration and checkpoint recycling.

Adaptive incremental checkpointing for data stream processing applications

A system to manage clustering vertices of a streaming graph includes a memory storing a computer program, a vertex structure storing vertices of the graph, and a cluster structure storing clusters of the graph, and a processor. Each cluster structure comprises a plurality of edge structures represent edges of the graph and each edge structure comprises two of the vertices. The processor is configured to execute the program to cluster the vertices using the structures and a maximum cluster size that is constant.

Evolution aware clustering of streaming graphs

A method for clustering vertices of streaming graphs includes: maintaining one or more clusters, assigning a random number to an incoming edge, computing a sampling threshold based on the current clusters, and adjusting the current clusters based on the random number and the sampling threshold.

Adaptive sampling schemes for clustering streaming graphs

Range query techniques are disclosed for use in accordance with data stream processing systems. A technique is provided for incrementally processing continual range queries against moving objects. This technique is applicable for location-aware services and applications. A technique for evaluating one or more continual range queries over one or more moving objects comprises maintaining a query index with one or more containment-encoded virtual constructs associated with the one or more continual range queries over the one or more moving objects, and incrementally evaluating the one or more continual range queries using the query index.

Range query methods and apparatus

A method for allocating parallel, independent, data tasks includes receiving data tasks, each of the data tasks having a penalty function, determining a generic ordering of the data tasks according to the penalty functions, wherein the generic ordering includes solving an aggregate objective function of the penalty functions, the method further including determining a schedule of the data tasks given the generic ordering, which packs the data tasks to be performed.

Scheduling parallel data tasks

Data analysis applications include model building components and stream processing components. To increase utility of the data analysis application, in one embodiment, the model building component of the data analysis application is managed. Management includes resource allocation and/or configuration adaptation of the model building component, as examples.

Managing resource allocation or configuration parameters of a model building component to build analytic models to increase the utility of data analysis applications

One embodiment of the present method and apparatus adaptive in-operator load shedding includes receiving at least two data streams (each comprising a plurality of tuples, or data items) into respective sliding windows of memory. A throttling fraction is then calculated based on input rates associated with the data streams and on currently available processing resources. Tuples are then selected for processing from the data streams in accordance with the throttling fraction, where the selected tuples represent a subset of all tuples contained within the sliding window.

Method and apparatus for adaptive in-operator load shedding

An embodiment of the invention provides a method for exploiting stateless and stateful data parallelism in a streaming application, wherein a compiler determines whether an operator of the streaming application is safe to parallelize based on a definition of the operator and an instance of the definition. The operator is not safe to parallelize when the operator has selectivity greater than 1, wherein the selectivity is the number of output tuples generated for each input tuple. Parallel regions are formed within the streaming application with the compiler when the operator is safe to parallelize. Synchronization strategies for the parallel regions are determined with the compiler, wherein the synchronization strategies are determined based on the definition of the operator and the instance of the definition. The synchronization strategies of the parallel regions are enforced with a runtime system.

Automatic exploitation of data parallelism in streaming applications

Techniques are disclosed for adaptive source filtering and load shedding in such data stream processing systems. For example, in one aspect of the invention, a method for use in filtering data in a distributed data stream processing system, wherein a server receives and processes one or more data streams from one or more data sources, comprises the steps of the server periodically re-configuring one or more filters and sending the one or more periodically re-configured filters to the one or more data sources, and the one or more data sources performing data filtering based on the one or more periodically re-configured filters received from the server.

Methods and apparatus for adaptive source filtering and load shedding for data stream processing

Techniques for composing an application resource model are disclosed. The techniques include obtaining operator-level metrics from an execution of a data stream processing application according to a first configuration, wherein the application is executed by nodes of the data stream processing system and the application includes processing elements comprised of multiple operators, wherein two or more of the operators are combined in a first combination to form a processing element according to the first configuration, generating operator-level resource functions from the first combination of operators based on the obtained operator-level metrics, and generating a processing element-level resource function using the generated operator-level resource functions to predict a model for the processing element formed by a second combination of operators, the processing element-level resource function representing an application resource model usable for predicting characteristics of the application executed according to a second configuration.

Application resource model composition from constituent components

Growing community of inventors

Yorktown Heights, NY, United States of America

Kun-Lung Wu