International Business Machines Corporation

Lawrence A Greene

Xiaoyan Pu

Yong Li

Eric Allen Jacobson

Yeh-Heng Sheng

Brian K Caufield

Ryan Pham

Shyam R Mudambi

Dong Jie Wei

A system includes at least one processor and processes an ETL job. The system analyzes a specification of the ETL job including one or more functional expressions to load data from one or more source data stores, process the data in memory, and store the processed data to one or more target data stores. One or more data flows are produced from the specification based on the one or more functional expressions. The one or more data flows utilize in-memory distributed data sets generated to accommodate parallel processing for loading and processing the data. The one or more data flows are optimized to assign operations to be performed on the one or more source data stores. The optimized data flows are executed to load the data to the one or more target data stores in accordance with the specification. Present invention embodiments further include methods and computer program products.

Transforming and loading data utilizing in-memory processing

A hybrid cluster environment with a public cloud cluster having nodes storing data and a plurality of private clusters is provided, wherein each of the plurality of private clusters has nodes storing data. Registration data that indicates a customer identifier, a new private cluster, and a file transfer server is received. The new private cluster is added to the plurality of private clusters in the hybrid cluster environment. Input to design a job to process data in the hybrid cluster environment is received. It is determined that the job is to be deployed to the new private cluster. The job is deployed to the new private cluster using the file transfer server, wherein the job is executed at the new private cluster. Job status information and one or more job logs are received with the file transfer server.

Data processing in a hybrid cluster environment

At least one application in a computing environment is executed and one or more performance metrics of the application are measured. The measured performance metrics are analyzed and an operational performance regression is detected. The detected operational performance regression is correlated with one or more recorded changes and the correlated changes are identified as a cause of the operational performance regression. The elements of the computing environment are alerted in accordance with the identified changes to adjust operational performance.

Detecting causes of performance regression to adjust data systems

Embodiments presented herein provide techniques for optimizing parallel data flows of a batch processing job using a profile of the processing job. An application retrieves a job profile for a processing job. The processing job has a plurality of processing stages specified in an execution profile. The job profile includes statistical data for at least one of the processing stages obtained during prior executions of the job. The application modifies properties of the execution profile based on the job profile to optimize the execution of the job. The application executes the processing job with the modified execution profile.

Dynamic optimization of workload execution based on statistical data collection and updated job profiling

Embodiments presented herein provide task management capabilities for designing a complex data integration workflow in an integrated design environment (IDE). A task management tool of the IDE allows a developer to tag various stages of a data integration workflow in a non-linear manner. When the task management tool receives a tag for a given stage, the task management tool identifies incomplete tasks associated with the stage and generates a task list that includes the incomplete tasks. The developer may return to completing any of the tasks in the workflow in any sequence as desired.

Task management integrated design environment for complex data integration applications

A request for analysis of a data integration job is received that includes one or more features and criteria for the analysis. Each feature is extracted from a job model representing the job by invoking a corresponding analytical rule for each feature. The analytical rule includes one or more operations and invoking the analytical rule performs the operations to analyze one or more job components associated with the corresponding feature as represented in the job model and to extract information pertaining to that feature.

Analysis of data integration job

The method of managing performance of data integration are described. A performance analyzer may receive data about a data integration job execution. The performance analyzer may determine whether there is a performance issue of the data integration job execution. The performance analyzer analyzes the data about the data integration job execution when there is a performance issue. The performance analyzer generates a job execution design recommendation based on the analysis of the data and a set of predefined recommendation rules. The performance analyzer then displays the data about the data integration job execution and when there is a generated job execution design recommendation, displaying the job execution design recommendation.

Generating and implementing data integration job execution design recommendations

Data flow disruptions over a series of data processing operators can be detected by a computer system that generates a profile for data flow at an operator. The profile can include data input, processing, and output wait times. Using the profile, the system can detect potential flow disruptions. If the potential disruption satisfies a rule, it is considered a data flow disruption and a recommendation associated with the satisfied rule is identified. The recommendation and the operator identity is displayed.

Detection of data flow bottlenecks and disruptions based on operator timing profiles in a parallel processing environment

According to one embodiment of the present invention, a system selectively compresses data fields in a parallel data flow. The system identifies within an execution plan for the parallel data flow a first instance of a data field within a stage of the parallel data flow. The system traces the identified data field through stages of the parallel data flow and determines a score value for the identified data field based on operations performed on the identified data field during traversal of the stages. The system compresses the identified data field based on the score value indicating a performance gain with respect to the compressed data field. Embodiments of the present invention further include a method and computer program product for selectively compressing data fields in a parallel data flow in substantially the same manners described above.

Growing community of inventors

Plainville, MA, United States of America

Lawrence A Greene