International Business Machines Corporation

Yong Li

Xiaoyan Pu

Eric Allen Jacobson

Lawrence A Greene

Jeff Jianfei Li

John Carroll Holmes

Shyam R Mudambi

Yeh-Heng Sheng

Ryan Pham

David Steven Sotkowitz

Jeffrey M Fischer

Ming Jiang

Krishna Kishore Bonagiri

Brian K Caufield

Eduardo N Spring

Ron E Liu

Rand K Barthel

Jean-Claude Mamou

Hanson Lieu

David T Meeks

Dong Jie Wei

Mining data transformation flows in spreadsheets includes identifying operations defined in a spreadsheet, identifying source data, in the spreadsheet, on which the operations operate, automatically creating an extract, transform, load (ETL) data transformation flow, and executing the created ETL data transformation flow. Creating the ETL data transformation flow includes selecting, in the ETL system, source data endpoint(s) for data extraction, selecting target data endpoint(s) for data loading, mapping at least one of the identified operations to ETL operation(s) for data transformation, and building the ETL data transformation flow, which defines extraction from the selected source data endpoint(s), transformation based on the ETL operation(s), and loading to the selected target data endpoint(s).

Mining data transformation flows in spreadsheets

A system includes at least one processor and processes an ETL job. The system analyzes a specification of the ETL job including one or more functional expressions to load data from one or more source data stores, process the data in memory, and store the processed data to one or more target data stores. One or more data flows are produced from the specification based on the one or more functional expressions. The one or more data flows utilize in-memory distributed data sets generated to accommodate parallel processing for loading and processing the data. The one or more data flows are optimized to assign operations to be performed on the one or more source data stores. The optimized data flows are executed to load the data to the one or more target data stores in accordance with the specification. Present invention embodiments further include methods and computer program products.

Transforming and loading data utilizing in-memory processing

A hybrid cluster environment with a public cloud cluster having nodes storing data and a plurality of private clusters is provided, wherein each of the plurality of private clusters has nodes storing data. Registration data that indicates a customer identifier, a new private cluster, and a file transfer server is received. The new private cluster is added to the plurality of private clusters in the hybrid cluster environment. Input to design a job to process data in the hybrid cluster environment is received. It is determined that the job is to be deployed to the new private cluster. The job is deployed to the new private cluster using the file transfer server, wherein the job is executed at the new private cluster. Job status information and one or more job logs are received with the file transfer server.

Data processing in a hybrid cluster environment

Provided are a computer program product, system, and method for generating a view for a schema. A schema provides a definition of elements, wherein at least one of the elements comprises a recursive element of a recursive data type, wherein the recursive data type allows for instances of the recursive element to include instances of that same recursive data type at different levels in a hierarchy. Indication is received to transform the recursive types to non-recursive structures. A view for the schema is generated including information on the indication to transform the recursive types to non-recursive structure, wherein the processing of the data defined by the view causes a parser to generate a structured element for each instance of the recursive element in schema data being parsed, wherein the generated structured element includes a unique identifier of the instance of the recursive element.

Generating a view for a schema including information on indication to transform recursive types to non-recursive structure in the schema

A mechanism is provided for applying matching data transformation information based on a user's editing of data within a document. User input identifying inputs provided by a user while editing a document within an application executing on the data processing system is received. A matching algorithm is executed based on user input to identify one or more candidate transformation operations within a transformation operation data structure that matches the user input. Responsive to failing to identify any candidate transformation operations, an indication is provided that no candidate transformation operations are identifiable. Responsive to one or more candidate transformation operations being identified, a list of transformation operations is provided that includes the one or more candidate transformation operations to the user via the data processing system.

Applying matching data transformation information based on a user's editing of data within a document

A user-defined function (UDF) is received in a central Computer System, which causes registration of the UDF and distributes the UDF to a cluster of computer system nodes configured for performing, in volatile memory of the nodes, extract-transform-load processing of data cached in the volatile memory of the nodes. First and second job specifications that include the UDF are received by the central Computer System, and the central computer system distributes instructions for the job specifications to the nodes including at least one instruction that invokes the UDF for loading and executing the UDF in the volatile memory of at least one of the nodes during runtime of the jobs. The central Computer System does not cause registration of the UDF again after receiving the first job specification.

Technology for extensible in-memory computing

Provided are a computer program product, system, and method for allocating physical nodes for processes in an execution plan. An execution plan is generated indicating a plurality of processes. A resource requirement is generated indicating requested physical nodes and an assignment of the processes to execute on the requested physical nodes. A determination is made from the resource requirement of a resource allocation of physical nodes for the requested physical nodes and the processes. The execution plan is updated to generate an updated execution plan indicating the physical nodes on which the processes will execute according to the received resource allocation.

Allocating physical nodes for processes in an execution plan

An approach for deploying workload in a multi-tenancy computing environment is provided. The approach generates, by one or more computer processors, a tenant ID and a plan ID for a tenant based, at least in part, on a tenant registration request. The approach stores, by one or more computer processors, the tenant ID and the plan ID into a shared system record. The approach receives, by one or more computer processors, a request to update a first tenant service plan. The approach determines, by one or more computer processors, one or more resource pools supporting a second tenant service plan based at least in part, on an association between the tenant ID and the plan ID. The approach deploys, by one or more computer processors, one or more resources from the one or more resource pools supporting the second tenant service plan.

Dynamic workload deployment for data integration services

At least one application in a computing environment is executed and one or more performance metrics of the application are measured. The measured performance metrics are analyzed and an operational performance regression is detected. The detected operational performance regression is correlated with one or more recorded changes and the correlated changes are identified as a cause of the operational performance regression. The elements of the computing environment are alerted in accordance with the identified changes to adjust operational performance.

Detecting causes of performance regression to adjust data systems

Timing parameters that influence an install time interval for installing a product on computing machines in a test environment in accordance with an installation configuration option are identified. A test value of the timing parameter and a test value of the install time are determined for each of the computing machines. The test values of the timing parameter and the install time determined for the sample computing machines are analyzed to determine an install time calculation expression for the installation configuration option. For installation in accordance with the installation configuration option in a normal operating environment, a current value of each of the timing parameters of the predetermined install time calculation expression for the installation configuration option. The install time interval in the normal operating environment is estimated based on the current value of the timing parameters and the install time calculation expression.

Accurately estimating install time

Embodiments presented herein provide techniques for optimizing parallel data flows of a batch processing job using a profile of the processing job. An application retrieves a job profile for a processing job. The processing job has a plurality of processing stages specified in an execution profile. The job profile includes statistical data for at least one of the processing stages obtained during prior executions of the job. The application modifies properties of the execution profile based on the job profile to optimize the execution of the job. The application executes the processing job with the modified execution profile.

Growing community of inventors

Newton, MA, United States of America

Yong Li