International Business Machines Corporation

Qi Jin

Sriram Srinivasan

Lin Xu

Hui Liao

Marion Else Behnen

Sriram Kolinjivadi Padmanabhan

Timo Pfahl

Dinkar Venkatraman Rao

Holger Pirk

Yannick Saillet

Richard Lee Cole

Hoi J Yoo

Muthukumar Thirunavukkarasu

A computer implemented method for generating code for an integrated data system. A mixed data flow is received. The mixed data flow contains mixed data flow operators, which are associated with multiple runtime environments. A graph is generated containing logical operators based on the mixed data flow in response to receiving the mixed data flow. The logical operators are independent of the plurality of runtime environments. The graph is converted to a model. The logical operators are converted to model operators associated with the multiple runtime environments. The model operators allow for analysis of operations for the mixed data flow. The model is converted into an execution plan graph. The execution plan graph is executable on different runtime environments.

Generating code for an integrated data system

Methods and apparatus, including computer program products, implementing and using techniques for integrating and data activities in a process flow. A data transformation activity is invoked through local or remote invocation. The data transformation activity is part of a process flow defined in a standard business process execution language format and is invoked from within the process flow. A system for executing a process flow including one or more control activities and one or more data transformation activities is also described. The system includes a process control engine for executing activities included in the process flow, a data transformation subsystem for storing domain specific definitions of data transformation processes of data in one or more databases, and a control data repository for storing domain specific activity information related to the process flow.

Supporting ETL processing in BPEL-based processes

Methods, systems, and computer program products for generating code from a data flow associated with an extract, transform, and load (ETL) process. In one implementation, the method includes identifying a data exchange requirement between a first operator and a second operator in the data flow. The first operator is a graphical object that represents a first data transformation step in the data flow and is associated with a first type of runtime engine, and the second operator is a graphical object that represents a second data transformation step in the data flow and is associated with a second type of runtime engine. The method further includes generating code to manage data staging between the first operator and the second operator in the data flow. The code exchanges data from a format associated with the first type of runtime engine to a format associated with the second type of runtime engine.

Modeling data exchange in a data flow of an extract, transform, and load (ETL) process

Methods, systems, and computer program products for generating code from a data flow associated with an extract, transform, and load (ETL) process. In one implementation, the method includes identifying a data exchange requirement between a first operator and a second operator in the data flow. The first operator is a graphical object that represents a first data transformation step in the data flow and is associated with a first type of runtime engine, and the second operator is a graphical object that represents a second data transformation step in the date flow and is associated with a second type of runtime engine. The method further includes generating code to manage data staging between the first operator and the second operator in the data flow. The code exchanges data from a format associated with the first type of runtime engine to a format associated with the second type of runtime engine.

Method and apparatus for modelling data exchange in a data flow of an extract, transform, and load (ETL) process

Methods and systems for implementing a splitter operation in an extract, transform, and load (ETL) process are provided. In one implementation, the method includes receiving a data flow including a splitter operation, and generating an execution plan graph based on the data flow. The execution plan graph includes structured query language (SQL) code for implementing the splitter operation, in which the structured query language (SQL) code is respectively executable among database servers associated with different vendors.

Method and apparatus for using set based structured query language (SQL) to implement extract, transform, and load (ETL) splitter operation

Methods, computer program products, and system for generating code for an extract, transform, and load (ETL) data flow are provided. In one implementation, the method includes receiving an ETL data flow representing a logical transformation and flow of data, placing a staging table at a pre-determined location in the ETL data flow to reduce a total number of staging tables required by the transformation, and generating code for the transformation based on the ETL data flow including the staging table placed at the pre-determined location.

Method and apparatus for generating code for an extract, transform, and load (ETL) data flow

A method, computer program product, and system for analyzing data using a data warehouse application are provided. The method, computer program product, and system provide for displaying a facet in a user interface of the data warehouse application, the facet classifying a plurality of documents, and displaying a facet attribute of the facet in the user interface of the data warehouse application, the facet attribute corresponding to a characteristic associated with each of the plurality of documents classified by the facet.

Data analysis using facet attributes

A method, computer program product, and system for integrating faceted and predefined multi-dimensional search interfaces are provided. The method, computer program product, and system provide for displaying a first set of results from a search request in a faceted search interface and a second set of results from the search request in a predefined multi-dimensional search interface, and responsive to a user action in the predefined multi-dimensional search interface affecting a scope of the search request, synchronizing the faceted search interface with changes in the predefined multi-dimensional search interface without requiring a separate user action in the faceted search interface.

Integration of predefined multi-dimensional and flexibly-ordered dynamic search interfaces

A computer implemented method for processing a mixed data flow. Mixed data flow operators are classified. The mixed data flow operators are associated with multiple runtime environments in which the mixed data flow operators can be executed. The mixed data flow operators are segregated into regions. A mixed data flow operator in a particular region is executable by one of the multiple runtime environments associated with the particular region. The execution of the regions is sequenced.

Classification and sequencing of mixed data flows

Methods, systems, and computer readable medium for displaying results of a search query. In one implementation, the method includes receiving a query, obtaining documents that satisfy the query, constructing a facet hierarchy based on documents that satisfy the query, creating a cube structure based on the facet hierarchy, and displaying a multi-dimensional search interface based on the cube structure.

Cube faceted data analysis

A computer implemented method, apparatus, and computer usable program code for generating an execution plan graph from a data flow. A metadata representation of the data flow is generated in response to receiving the data flow. A set of code units is generated from the metadata representation. Each code unit in the set of code units is executable on multiple different types of runtime engines. The set of code units is processed to produce the execution plan graph.

Data flow system and method for heterogeneous data integration environments

A technique for processing input data is provided. Multiple input rows to beloaded into a first structure are received. Each input row of the multiple input rows is processed to classify each input row as one of an insert row and an update row, wherein input duplicates are stored in the first structure and index entries for the input duplicates are stored in a second structure. After the multiple input rows have been processed, the input duplicates are automatically re-applied to the first structure and the index entries stored in the second structure are processed.

Method, system, and article of manufacture for processing updates to insert operations

A technique is provided for loading input data in one or more hierarchical format input files into a data store. Parallel processing of one or more input files is performed to output data. The data is serially loaded into the data store while enforcing the order of the data in the one or more input files. Each input file may be a physical file or a logical file.

Growing community of inventors

San Jose, CA, United States of America

Qi Jin