International Business Machines Corporation

Sriram Kolinjivadi Padmanabhan

Manish Anand Bhide

Timothy Ray Malkemus

Bishwaranjan Bhattacharjee

Manoj Kumar Agarwal

Srinivas Kiran Mittapalli

Rajeev Gupta

George Andrei Mihaila

Himanshu Gupta

Sanjeev K Gupta

Mukesh Kumar Mohania

Sam Sampson Lightstone

Matthew Albert Huras

Min Wang

Ritesh Kumar Gupta

Brian K Caufield

Prasan Roy

Ron E Liu

Leslie Anne Cranston

Prathima Nagaraj

Nianjun Zhou

Yuan-Chi Chang

Dikran S Meliksetian

Prasad Manikarao Deshpande

Christian Alexander Lang

Qi Jin

Atreyee Dey

Hui Liao

Chung-Sheng Li

John Richard Smith

Philip Shilung Yu

Lin Xu

Adam J Storm

Oded Shmueli

Rajesh Ramkrishna Bordawekar

James D Spyker

Xiaoyan Pu

Michael Jeffrey Winer

Alexander Gorelik

Daniele Costante Zilio

Ramesh Chandra Agarwal

Girish Venkatachaliah

Shyam R Mudambi

Catherine Suzanne McArthur

Anant Deep Jhingran

Salil Joshi

Mi Wan Shum

DongJie Wei

Chun H Sun

Tony Wen Hsun Lai

Richard E Swagerman

Vivek S Tirumalaraju

Kwai Wong

Charu Chandra Aggarwal

Kun-Lung Wu

Sachindra Joshi

Charles Philippe Tresser

Bruce Gilbert Lindsay

Renu Tewari

Kenneth Alan Goldman

Yi Chen

Milind R Naphade

Lawrence David Bergman

Hui-I Hsiao

L Venkata Subramaniam

Lipyeow Lim

John Steven Langford

Sriram Raghavan

Matthew Leon Hill

David Mun-Hien Choy

Ioana Roxana Stanoi

Tanveer Afzal Faruquie

Naoki Abe

Marion Else Behnen

Hima Prasad Karanam

Sriram Srinivasan

Jy-Jine James Lin

Amir Bar-Or

Matthias Gruetzner

Murray Scott Campbell

Chidanand V Apte

Gene Y C Fuh

Lubor J Kollar

Robin Williams

Khalil S Amiri

Kaarel Truuvert

Dinkar Venkatraman Rao

Sanghyun Park

Carl E Abrams

Jingmin He

Petrus K C Chan

Sebastian Ertel

Srilakshmi Kotwal

Matthew A Hilbert

Kathleen M Troidle

Khalil S c/o Ibm United Kingdom Limited Amiri

Sanghyun c/o Ibm United Kingdom Limited Park

A computer-implemented method includes inserting a bloom filter creation stage after an inner data source identification stage, wherein a join operation is to be performed to join an outer data source with the inner data source. The method inserts a bloom filter search stage after an outer data source identification stage, wherein each row of data from the outer data source is searched against a bloom filter for the inner data source during the bloom filter search stage. The method initializes a read on the inner data source. Subsequent to determining the bloom filter creation stage is complete, the method initializes a read on the outer data source. The method performs the join operation at a join stage.

Bloom filter utilization for join processing

Entity mappings that produce matching entities for a first data asset having attributes and a second data asset having attributes are generated by: generating entity mappings that produce matching entities for a first data asset having attributes with attribute values and a second data asset having attributes with attribute values by: matching the attribute values of the attributes of the first data asset with the attribute values of the attributes of the second data asset, using the matching attribute values to generate matching attribute pairs, and using the matching attribute pairs to identify entity mappings; computing an entity mapping score for each of the entity mappings based on a combination of factors; ranking the entity mappings based on each entity mapping score; and using some of the ranked entity mappings to determine whether a same real-world entity is described by the first data asset and the second data asset.

Identifying entity mappings across data assets

Methods and arrangements for identifying related data in different data sets to assist in searching the data sets. A first data asset and a second data asset are accessed. Common entities are identified between the first and second data assets. A score is determined for the relationship between the first and second data assets, based on the identified common entities. One or more relationship contexts are determined for the relationship between the first and second data assets, and the relationship score and one or more relationship contexts are used to join at least a portion of each of the first and second data assets as a basis for subsequent searching. Other variants and embodiments are broadly contemplated herein.

Contextualization of entity relationships

Provided are techniques for multiple stage workload management. A staging queue and a run queue are provided. A workload is received. In response to determining that application resources are not available and that the workload has not been previously semi-started, the workload is added to the staging queue. In response to determining that the application resources are not available and that the workload has been semi-started, and, in response to determining that run resources are available, the workload is started. In response to determining that the application resources are not available and that the workload has been semi-started, and, in response to determining that the run resources are not available, adding the workload to the run queue.

Multiple stage workload management system

A computer implemented method for generating code for an integrated data system. A mixed data flow is received. The mixed data flow contains mixed data flow operators, which are associated with multiple runtime environments. A graph is generated containing logical operators based on the mixed data flow in response to receiving the mixed data flow. The logical operators are independent of the plurality of runtime environments. The graph is converted to a model. The logical operators are converted to model operators associated with the multiple runtime environments. The model operators allow for analysis of operations for the mixed data flow. The model is converted into an execution plan graph. The execution plan graph is executable on different runtime environments.

Generating code for an integrated data system

A computer determines a business glossary, wherein the business glossary includes one or more business terms that correspond to one or more assets. The computer receives a request to create an ETL job associated with a business term. The computer creates an ETL job for the business term, wherein the ETL job includes a business glossary stage.

Extracting, transforming, and loading (ETL) job creation using business term stage

Systems and associated methods for address standardization and applications related thereto are described. Embodiments exploit a common context in a taxonomy and a given address to detect and correct deviations in the address. Embodiments establish a possible path from a root of the taxonomy to a leaf in the taxonomy that can possibly generate a given address. Given a new address, embodiments use complete addresses, and/or segments or elements thereof, to compute the representations of the elements and find a closest matching leaf in the taxonomy. Embodiments then traverse the path to a root node to detect the agreement and disagreement between the path and the address entry. Taxonomical structured is thus used to detect, segregate and standardize the expected fields.

Systems and methods for standardization and de-duplication of addresses using taxonomy

A computer software implemented method for transforming a first extract transform load (ETL) job having at least some unload transform load (UTL) portions. The method includes the following steps: (i) decomposing the first ETL job into an intermediate set of one or more jobs; and (ii) for each job of the intermediate set, transforming the job into a transactionally equivalent job to yield a final set of one or more jobs. The decomposing is performed so that each job of the intermediate jobs set is a Simple UTL job. The transforming is performed so that each job of the final set includes no UTL portions.

Creation of change-based data integration jobs

Provided are techniques for sharing a partitioned data set across parallel applications. Under control of a producing application, a partitioned data set is generated; a descriptor that describes the partitioned data set is generated; and the descriptor is registered in a registry. Under control of a consuming application, the registry is accessed to obtain the descriptor of the partitioned data set; and the descriptor is uses to determine how to process the partitioned data set.

Sharing a partitioned data set across parallel applications

An embodiment provides intermediate data derived in the form of column stores which are in turn based on hierarchical data stores. This intermediate data represents a reduced subset of data matched appropriately to a query (or modified query) such that the amount of data handled in a query processing task on large data is greatly reduced. An embodiment may appropriately choose column data stores and/or modify queries in order leverage parallelization techniques such as map-reduce in order to query large data. The result is the ability to query large data stores in parallel while reducing the amount of data that must be handled.

System and a method for hierarchical data column storage and efficient query processing

A method, a computer program product and a system identify partition locations within an extended markup language (XML) document without parsing so as to process portions of said document in parallel. The XML document includes sections required to remain continuous. The document is scanned for continuous sections without parsing, and boundaries of the initial partitions are adjusted to reside outside the continuous sections to determine resulting partitions for the document. The resulting partitions may be processed in parallel to provide the document information for storage.

Finding partition boundaries for parallel processing of markup language documents

A system for receiving a declarative specification including a plurality of stages. Each stage specifies an atomic operation, a data input to the atomic operation, and a data output from the atomic operation. The data input is characterized by a data type. Links between at least two of the stages are generated to create a data integration workflow. The data integration workflow is compiled to generate computer code for execution on a parallel processing platform. The computer code configured to perform at least one of data preparation and data analysis.

Declarative specification of data integration workflows for execution on parallel processing platforms

Growing community of inventors

San Jose, CA, United States of America

Sriram Kolinjivadi Padmanabhan