International Business Machines Corporation

Hui Liao

Qi Jin

Lin Xu

Sriram Kolinjivadi Padmanabhan

Sriram Srinivasan

Marion Else Behnen

Dinkar Venkatraman Rao

A computer implemented method for generating code for an integrated data system. A mixed data flow is received. The mixed data flow contains mixed data flow operators, which are associated with multiple runtime environments. A graph is generated containing logical operators based on the mixed data flow in response to receiving the mixed data flow. The logical operators are independent of the plurality of runtime environments. The graph is converted to a model. The logical operators are converted to model operators associated with the multiple runtime environments. The model operators allow for analysis of operations for the mixed data flow. The model is converted into an execution plan graph. The execution plan graph is executable on different runtime environments.

Generating code for an integrated data system

Methods, systems, and computer program products for generating code from a data flow associated with an extract, transform, and load (ETL) process. In one implementation, the method includes identifying a data exchange requirement between a first operator and a second operator in the data flow. The first operator is a graphical object that represents a first data transformation step in the data flow and is associated with a first type of runtime engine, and the second operator is a graphical object that represents a second data transformation step in the data flow and is associated with a second type of runtime engine. The method further includes generating code to manage data staging between the first operator and the second operator in the data flow. The code exchanges data from a format associated with the first type of runtime engine to a format associated with the second type of runtime engine.

Modeling data exchange in a data flow of an extract, transform, and load (ETL) process

Methods, systems, and computer program products for generating code from a data flow associated with an extract, transform, and load (ETL) process. In one implementation, the method includes identifying a data exchange requirement between a first operator and a second operator in the data flow. The first operator is a graphical object that represents a first data transformation step in the data flow and is associated with a first type of runtime engine, and the second operator is a graphical object that represents a second data transformation step in the date flow and is associated with a second type of runtime engine. The method further includes generating code to manage data staging between the first operator and the second operator in the data flow. The code exchanges data from a format associated with the first type of runtime engine to a format associated with the second type of runtime engine.

Method and apparatus for modelling data exchange in a data flow of an extract, transform, and load (ETL) process

Methods and systems for implementing a splitter operation in an extract, transform, and load (ETL) process are provided. In one implementation, the method includes receiving a data flow including a splitter operation, and generating an execution plan graph based on the data flow. The execution plan graph includes structured query language (SQL) code for implementing the splitter operation, in which the structured query language (SQL) code is respectively executable among database servers associated with different vendors.

Method and apparatus for using set based structured query language (SQL) to implement extract, transform, and load (ETL) splitter operation

Methods, computer program products, and system for generating code for an extract, transform, and load (ETL) data flow are provided. In one implementation, the method includes receiving an ETL data flow representing a logical transformation and flow of data, placing a staging table at a pre-determined location in the ETL data flow to reduce a total number of staging tables required by the transformation, and generating code for the transformation based on the ETL data flow including the staging table placed at the pre-determined location.

Method and apparatus for generating code for an extract, transform, and load (ETL) data flow

A computer implemented method, apparatus, and computer usable program code for generating an execution plan graph from a data flow. A metadata representation of the data flow is generated in response to receiving the data flow. A set of code units is generated from the metadata representation. Each code unit in the set of code units is executable on multiple different types of runtime engines. The set of code units is processed to produce the execution plan graph.

Growing community of inventors

Santa Clara, CA, United States of America

Hui Liao