International Business Machines Corporation

Ritesh Kumar Gupta

Namit Kabra

Raghavendra Rao Dhayapule

Sumit Kumar

Rajesh Phillips

Vijay Ekambaram

Sriram Kolinjivadi Padmanabhan

Smitkumar Narotambhai Marvaniya

Eric Allen Jacobson

Prathima Nagaraj

Scott Louis Brokaw

Yannick Saillet

Krishna Kishore Bonagiri

Sameep Mehta

Manish Kesarwani

Ajay Kumar Sood

Bijo A Thomas

Jo Arao Ramos

Likhitha Maddirala

Niels Dominic Pardon

Madhusmita P Patil

Anton Zorin

Akshay Gautam

Arundhathi Erla

Swetha Patil

Ramesh Rajagopalan

Kalyanji Chintakayala

Venkat R Arra

Venkata R Somisetty

Santosh K Janumahanthi

Nagesh M V G V

Praveen Thota

Vikram K Vb

Indrani Ghatare

Ron Reuben

Embodiments for identifying an optimal cloud computing environment for a computing task is disclosed. Embodiments comprises receiving a computing task to be executed in a cloud computing environment, wherein the computing task requires a set of cloud computing environment parameter values of the cloud computing environment, pre-selecting a set of candidate cloud computing environments, each of which meets the set of cloud computing environment parameter values, ranking the candidate cloud computing environments using reward-based ranking parameter values of the candidate cloud computing environments as an additional selection constraint, and selecting the highest ranking cloud computing environment as the optimal cloud computing environment for the computing task. Furthermore, embodiments comprise executing the computing task in the optimal cloud computing environment, monitoring execution when executing the computing task, and updating parameter values of the reward-based ranking for the selected optimal cloud computing environment.

Selecting best cloud computing environment in a hybrid cloud scenario

The method performs at the orchestration interface at which update information, including changes to tasks of a workflow, is received from a task manager system (TMS), where the workflow includes a set of tasks, inputs to the tasks, and outputs from the tasks. The inputs and outputs determine runtime dependencies between the tasks. Based on the update information received, the orchestration interface populates a topology of nodes and edges as a directed acyclic graph (DAG) that maps nodes to tasks and edges to runtime dependencies between tasks, based on node inputs and outputs. The orchestration interface instructs the execution of the tasks and handling dependencies by interacting with a task execution system (TES) and by traversing the DAG, the orchestration interface identifies tasks that depend on completed tasks as per the runtime dependencies and instructs the TES to execute the dependent tasks identified.

Automatically orchestrating a computerized workflow

Targeted acquisition of data for model training includes identifying attributes of classified samples of a collection of samples classified by a classification model, and generating at least one query based on the identified attributes, the at least one query tailored, based on the attributes, to retrieve additional training data for training the classification model to more accurately classify samples and avoid incorrect sample classification.

Targeted data acquisition for model training

Targeted acquisition of data for model training includes automatically generating metadata describing samples, of an initial dataset, in neighborhoods of an embedding space in which the samples are embedded. The samples described by the automatically generated metadata are classified by a classification model, and include both correctly classified samples in the neighborhoods and incorrectly classified samples in the neighborhoods. Additionally, attributes of one or more correctly classified samples of the collection of samples and one or more incorrectly classified samples of the collection of samples are identified, and queries are generated based on the identified attributes, the queries tailored, based on the attributes, to retrieve additional training data for training the classification model to more accurately classify samples and avoid incorrect sample classification.

A method, computer system, and a computer program product for data pipeline prioritization is provided. Embodiments may include receiving, by a cognitive rules engine, one or more data pipelines. Embodiments may then include analyzing, using a computational method of the cognitive rules engine, the one or more data pipelines. Embodiments may lastly include prioritizing the one or more data pipelines based on a result of the computational method of the cognitive rules engine.

Prioritization and prediction of jobs using cognitive rules engine

An approach is provided for completing a pipeline graph. Using a deep learning based sequence model, an initial data pipeline having a sequence of nodes is generated. Mismatch(es) between data formats required by input and output in the sequence of nodes is identified. Virtual gap node(s) that correct the mismatch(es) are added to the initial data pipeline. For a given virtual gap node, tentative graph structures are determined using knowledge graphs and a crowd sourced validation system. Reuse forecast scores and performance scores for the tentative graph structures are calculated. Based on the reuse forecast scores and the performance scores, a final graph structure for implementing the given virtual gap node is determined.

Automatically completing a pipeline graph in an internet of things network

A computer-implemented method applies insights from a variety of data sources to each of the data sources. The method includes identifying a set of data sources, wherein each of the data sources are associated with a domain. The method includes analyzing documentation for each of the data sources. The method further includes extracting a set of attributes for each data source, and determining a data schema associated with each data source. The method includes mapping each data schema to a common domain schema. The method also includes linking, based on the mapping and on the set of attributes for each data source, common features across each data source. The method includes generating, in response to the linking, a knowledge graph. The method further includes preparing a visual display for a set of domain insights; and forking the set of domain insights into a first data source.

Insight expansion in smart data retention systems

In an approach to dynamically identifying and modifying the parallelism of a particular task in a pipeline, the optimal execution time of each stage in a dynamic pipeline is calculated. The actual execution time of each stage in the dynamic pipeline is measured. Whether the actual time of completion of the data processing job will exceed a threshold is determined. If it is determined that the actual time of completion of the data processing job will exceed the threshold, then additional instances of the stages are created.

Dynamically modifying the parallelism of a task in a pipeline

A self-learning computer-based system has access to multiple runtime modules that are each capable of performing a particular algorithm. Each runtime module implements the algorithm with different code or runs in a different runtime environment. The system responds to a request to run the algorithm by selecting the runtime module or runtime environment that the system predicts will provide the most desirable results based on parameters like accuracy, performance, cost, resource-efficiency, or policy compliance. The system learns how to make such predictions through training sessions conducted by a machine-learning component. This training teaches the system that previous module selections produced certain types of results in the presence of certain conditions. After determining whether similar conditions currently exist, the system uses rules inferred from the training sessions to select the runtime module most likely to produce desired results.

Self-learning selection of information-analysis runtimes

A method includes: receiving, by a computer device, resource request for a data integration job, wherein the resource request is received from a job executor module and defines processes of the data integration job; allocating, by the computer device, containers for the processes of the data integration job; launching, by the computer device, a respective wrapper script on each respective one of the containers after allocating the respective one of the containers; and transmitting, by the computer device and in response to the allocating, node details to the job executor module. In embodiments, the wrapper script running on the container is configured to repeatedly check a predefined location for process commands from a job executor. After the resource manager allocates all the containers for a data integration job according to a resource request, the job executor writes the process commands to the predefined location. Each wrapper script continues to check the predefined location for the process command that it is assigned to run, and runs the process command as soon as it is available at the predefined location. The process commands may be indexed with index values matching those assigned to respective ones of the wrapper scripts.

Handling expiration of resources allocated by a resource manager running a data integration job

A set of resources required to process a data integration job is determined. In response to determining that the set of resources is not available, queue occupation, for each queue in the computing environment, is predicted. Queue occupation is a workload of queue resources for a future time based on a previous workload. A best queue is selected based on the predicted queue occupation. The best queue is the queue or queues in the computing environment available to be assigned to process the data integration job without preemption. The data integration job is processed using the best queue. It is determined whether a preemption event occurred causing the removal of resources from the best queue. A checkpoint is created in response to determining that a preemption event occurred. The checkpoint indicates the last successful operation completed and provides a point where processing can resume when resources become available.

Resolving container preemption

A method and system for improving data and memory reorganization and storage technology is provided. The method includes configuring data capture and analysis settings of a database system resulting in configured data capture settings. A data and associated memory analysis request is received and specified test code is selected. A specified portion of data and associated memory is selected and the specified analysis code is executed resulting in execution of said specified type of analysis with respect to the specified portion of said data and associated memory. The specified portion of said data and associated memory is modified and stored.

Data and memory reorganization

A computer detects initialization of a first messaging session. The computer receives one or more message commands relating to one or more messages. The computer monitors message progress of the first messaging session. The computer detects errors in the message progress. The computer starts one or more additional messaging session(s). The computer monitors message progress of the first messaging session and the one or more additional messaging session(s). The computer detects contraction of messaging sessions is appropriate. The computer closes one or more messaging sessions.

Dynamic communication session management

A method for testing an extract, transform, load job (ETL job) is provided. The method may include receiving an uploaded ETL job. The method may also include executing a functionality test of the uploaded ETL job using a test bucket. The method may further include determining the executed functionality test produces a failure result. The method may also include creating at least one test sub-job corresponding to at least one stage within on the uploaded ETL job. The method may further include generating a plurality of test data for each of the at least one created test sub-job. The method may also include executing the at least one created test sub-job using the plurality of generated test data. The method may further include creating a report based on a plurality of results associated with the at least one executed test sub-job.

Growing community of inventors

Hyderabad Telangana, India

Ritesh Kumar Gupta