International Business Machines Corporation

Krishna Kishore Bonagiri

Eric Allen Jacobson

Kalyanji Chintakayala

Manish Anand Bhide

Yannick Saillet

Yong Li

Xiaoyan Pu

Namit Kabra

Mike W Grasselt

Srinivas Kiran Mittapalli

Ritesh Kumar Gupta

Rajesh K Pandey

Nagaraju Narla

Sumit Negi

Scott Louis Brokaw

Ron E Liu

Indrani Ghatare

Juan M George

A database of deployed configurations, as well as attempted configurations that failed is maintained and used as reference to compare against configurations of attempted software deployments. Upon detecting a failed deployment, disclosed embodiments search the database for working configurations that most closely resemble the failed configuration, and rank the configurations based on various criteria. Disclosed embodiments may then automatically select a highest ranked working configuration, and perform an automatic upgrade of the necessary components to create a working configuration.

Automatic application dependency management

A method provides for classifying data fields of a dataset. A classifier configured for determining confidence values for a plurality of data classes for the data fields may be applied. Using the confidence values, data class candidates may be identified. Data fields may be determined for which a plurality of data class candidates is identifiable. Using previous user-selected data class assignments, a probability may be determined for the data class candidates that the respective data class candidate is a data class to which the respective data field is to be assigned. The data fields may be classified using the probabilities to select for the data fields a data class from the data class candidates. The dataset may be provided with metadata identifying for the data fields the data classes to which the respective data fields are assigned.

Data classification

Techniques are described relating to automatic data standardization in a managed services domain of a cloud computing environment. An associated computer-implemented method includes receiving a dataset during a data onboarding procedure and classifying datapoints within the dataset. The method further includes applying a machine learning data standardization model to each classified datapoint within the dataset and deriving a proposed set of data standardization rules for the dataset based upon any standardization modification determined consequent to model application. Optionally, the method includes presenting the proposed set of data standardization rules for client review and, responsive to acceptance of the proposed set of data standardization rules, applying the proposed set of data standardization rules to the dataset. The method further includes, responsive to acceptance of the proposed set of data standardization rules, updating the machine learning data standardization model accordingly.

Standardization in the context of data integration

A method includes: receiving, by a computer device, resource request for a data integration job, wherein the resource request is received from a job executor module and defines processes of the data integration job; allocating, by the computer device, containers for the processes of the data integration job; launching, by the computer device, a respective wrapper script on each respective one of the containers after allocating the respective one of the containers; and transmitting, by the computer device and in response to the allocating, node details to the job executor module. In embodiments, the wrapper script running on the container is configured to repeatedly check a predefined location for process commands from a job executor. After the resource manager allocates all the containers for a data integration job according to a resource request, the job executor writes the process commands to the predefined location. Each wrapper script continues to check the predefined location for the process command that it is assigned to run, and runs the process command as soon as it is available at the predefined location. The process commands may be indexed with index values matching those assigned to respective ones of the wrapper scripts.

Handling expiration of resources allocated by a resource manager running a data integration job

A discussion thread is associated with a master recipients list that is maintained at an author's computer or a server computer. The recipients list is revised as participants in the discussion are added to and/or removed from the discussion. An author is notified when target recipients do not match the master recipients list. When a participant is added to the discussion from an old discussion instance, a thread manager provides for the added participant to receive a more current discussion instance.

Management of parallel user inputs in electronic communications

A set of resources required to process a data integration job is determined. In response to determining that the set of resources is not available, queue occupation, for each queue in the computing environment, is predicted. Queue occupation is a workload of queue resources for a future time based on a previous workload. A best queue is selected based on the predicted queue occupation. The best queue is the queue or queues in the computing environment available to be assigned to process the data integration job without preemption. The data integration job is processed using the best queue. It is determined whether a preemption event occurred causing the removal of resources from the best queue. A checkpoint is created in response to determining that a preemption event occurred. The checkpoint indicates the last successful operation completed and provides a point where processing can resume when resources become available.

Resolving container preemption

Improving allocation of network resources by receiving node names for resource allocation, checking a bookmark file of bad nodes for the received node names, selecting good nodes from the received nodes for command execution, sending commands to selected good nodes, identifying bad nodes during command execution; and adding the identified bad nodes to the bookmark file.

Resource allocation for data integration

Data classification includes tracking classification of columns of data into data classes of a collection of classes available for classifying the columns, obtaining a target column of data, of a target dataset, to be classified into a data class of the collection of candidate classes, and classifying the target column of data into a data class of the collection of classes based on historical data classification characteristics provided by the tracking. The classifying includes selecting a group of candidate data classes of the collection of classes to compare to value(s) of the target column, the selecting excludes at least some candidate data classes of the collection from comparison to the value(s), and establishing a priority between the candidate data classes of the group of candidate classes in comparing the value(s) of the target column of data to the selected group of candidate classes.

Disclosed aspects relate to data integration application execution management using both a set of managed nodes and a set of unmanaged nodes. An execution request for a data integration application may be sensed. A first set of commands of the data integration application may be assigned to the set of managed nodes. A second set of commands of the data integration application may be assigned to the set of unmanaged nodes. Execution of the data integration application may be initiated using the set of managed nodes to process the first set of commands and the set of unmanaged nodes to process the second set of commands.

Data integration application execution management

Provided are a computer program product, system, and method for allocating physical nodes for processes in an execution plan. An execution plan is generated indicating a plurality of processes. A resource requirement is generated indicating requested physical nodes and an assignment of the processes to execute on the requested physical nodes. A determination is made from the resource requirement of a resource allocation of physical nodes for the requested physical nodes and the processes. The execution plan is updated to generate an updated execution plan indicating the physical nodes on which the processes will execute according to the received resource allocation.

Allocating physical nodes for processes in an execution plan

Performing an extract, transform, and load (ETL) process. Column data is received by a stage of the ETL process. The size of the received column data is ascertained by the stage. In response to determining that the size of the column data exceeds a predefined threshold, the stage saves the column data and creates a data locator associated with the column data. The created data locator advances through successive downstream stages of the ETL process as a replacement for the column data.

Growing community of inventors

Ambajipet, India

Krishna Kishore Bonagiri