International Business Machines Corporation

Torrent Systems, Inc.

Shyam R Mudambi

Yong Li

Xiaoyan Pu

Eric Allen Jacobson

Manish Anand Bhide

Michael James Beckerle

Allen M Razdow

Jerry Lee Callen

Steven Rosenthal

David Albert Blair

Lawrence A Bookman

Sriram Kolinjivadi Padmanabhan

Vivek S Tirumalaraju

Robert Louis Krawitz

Jean-Claude Mamou

Brian K Caufield

Lawrence A Greene

Dong Jie Wei

Robert Louise Krawitz

A computer-implemented method includes inserting a bloom filter creation stage after an inner data source identification stage, wherein a join operation is to be performed to join an outer data source with the inner data source. The method inserts a bloom filter search stage after an outer data source identification stage, wherein each row of data from the outer data source is searched against a bloom filter for the inner data source during the bloom filter search stage. The method initializes a read on the inner data source. Subsequent to determining the bloom filter creation stage is complete, the method initializes a read on the outer data source. The method performs the join operation at a join stage.

Bloom filter utilization for join processing

A computer manages methods for utilizing an index to manage access to data in a dataset stored in one or more file locations in an ETL tool by receiving a request to access a dataset associated with one or more file locations, wherein the dataset is stored in the one or more file locations. The computer queries an index for the one or more file locations associated with the dataset, wherein the dataset has another index for data in the dataset. The computer receives the one or more file locations associated with the dataset. The computer determines to cache the request to access the one or more file locations for the dataset until one or more thresholds are met, wherein the cached request is part of a total number of cached requests.

Index utilization in ETL tools

System, method, and computer program product to process parallel computing tasks on a distributed computing system, by computing an execution plan for a parallel computing job to be executed on the distributed computing system, the distributed computing system comprising a plurality of compute nodes, generating, based on the execution plan, an ordered set of tasks, the ordered set of tasks comprising: (i) configuration tasks, and (ii) execution tasks for executing the parallel computing job on the distributed computing system, and launching a distributed computing application to assign the tasks of the ordered set of tasks to the plurality of compute nodes to execute the parallel computing job on the distributed computing system.

Deploying parallel data integration applications to distributed computing environments

System, method, and computer program product to perform an operation for task-based modeling for parallel data integration, by determining, for a data flow, a set of processing units, each of the set of processing units defining one or more data processing operations to process the data flow, generating a set of tasks to represent the set of processing units, each task in the set of tasks comprising one or more of the data processing operations of the set of processing units, optimizing the set of tasks based on a set of characteristics of the data flow, and generating a composite execution plan based on the optimized set of tasks to process the data flow in a distributed computing environment.

Task-based modeling for parallel data integration

Data flow disruptions over a series of data processing operators can be detected by a computer system that generates a profile for data flow at an operator. The profile can include data input, processing, and output wait times. Using the profile, the system can detect potential flow disruptions. If the potential disruption satisfies a rule, it is considered a data flow disruption and a recommendation associated with the satisfied rule is identified. The recommendation and the operator identity is displayed.

Detection of data flow bottlenecks and disruptions based on operator timing profiles in a parallel processing environment

Techniques are disclosed for data integration on retargetable engines in a networked environment. The networked environment includes data processing engines of different types and having different sets of characteristics. A request is received execute a data flow model in the networked environment. The data flow model includes data flow objects. A first data processing engine is programmatically selected based on a predefined set of criteria and the sets of characteristics of the data processing engines. The data flow model is executed using the selected data processing engine and responsive to the request.

Data integration on retargetable engines in a networked environment

Techniques are disclosed for qualified checkpointing of a data flow model having data flow operators and links connecting the data flow operators. A link of the data flow model is selected based on a set of checkpoint criteria. A checkpoint is generated for the selected link. The checkpoint is selected from different checkpoint types. The generated checkpoint is assigned to the selected link. The data flow model, having at least one link with no assigned checkpoint, is executed.

Selective checkpointing of links in a data flow based on a set of predefined criteria

With a continuous source of data relating to transactions, the data may be segmented and processed in a data flow arrangement, optionally in parallel, and the data may be processed without storing the data in an intermediate database. Data from multiple sources may be processed in parallel. The segmentation also may define points at which aggregate outputs may be provided, and where checkpoints may be established.

Growing community of inventors

Wayland, MA, United States of America

Shyam R Mudambi