International Business Machines Corporation

Cisco Technology, Inc.

Intuit, Inc.

Kyndryl, Inc.

Himanshu Gupta

Mukesh Kumar Mohania

Sameep Mehta

Rajeev Gupta

Prasan Roy

Sriram Kolinjivadi Padmanabhan

Rajmohan Chandrahasan

Manish Anand Bhide

Manoj Kumar Agarwal

Nitin Kumar Gupta

Sanjeev K Gupta

Sriram Raghavan

Pranay Kumar Lohia

Ullas Balan Nambiar

Venkat Chakravarthy

Scott Schumacher

Samiulla Zakir Hussain Shaikh

Sriram Lakshminarasimhan

Tanveer A Faruquie

Bhupesh S Chawda

Srikanth Govindaraj Tamilselvam

Amith Singhee

L Venkata Subramaniam

Vijay Arya

Andrew T Penrose

Akshar Kaul

Bala Dutt

Manish Kumar

Andrzej Jan Wrobel

Laurent Sebastien Mignet

Atul S Kumar

Jitendra Singh

Stuart A Siegel

Manish R Shah

Vaibhav S Dantale

Anuj Kumar Garg

Harsha K Navada

Emma Rose Tucker

Neeraj Kumar Gupta

Manoj K

Venkatesh Basappa Neldurg

Ayaz Nabi

Srikar B S

Saket Gurukar

Rishabh Gupta

Ravi Vishnu Israni

Bhanu Mudhireddy

Methods, systems, and computer program products for identifying anomalous transformations using lineage data are provided herein. A computer-implemented method includes generating a set of column profiles for a corresponding set of columns within one or more datasets based at least in part on lineage data and glossary data, wherein the lineage data comprises information related to transformations performed on each column in the set by a computing platform, and wherein the glossary data comprises information related to one or more terms assigned to one or more of the columns; obtaining information related to a new transformation involving at least one column in the set of columns; comparing the new transformation to the set of column profiles to determine whether the new transformation is anomalous; and in response to determining the new transformation is anomalous, outputting an alert to a user of the computing platform.

Identifying anomalous transformations using lineage data

A computer implemented method creates microservices for an application. A number of processor units clusters programs and data structures for the application using runtime metadata to form groups of the programs and data structures. The runtime metadata is obtained from running the application. The number of processor units creates a design for the microservices for the application using the groups of the programs and the data structures.

Microservice creation using runtime metadata

One embodiment provides a method for recommending model characteristics to be used in developing a target geo-spatial physical model for a target geographic location utilizing historical lineage data corresponding to historical geo-spatial physical models, including: receiving information related to the target geographic location, wherein the information describes geographical and domain features of the target geographic location; identifying, using at least one similarity algorithm, at least one other geographic location that is similar to the target geographic location, wherein the at least one geographic location has at least one corresponding historical geo-spatial physical model; and recommending, using at least one machine-learning model and based upon the at least one other geographic location, initial model characteristics for developing and deploying the target geo-spatial physical model.

Development of geo-spatial physical models using historical lineage data

An example operation includes one or more of identifying, by an archiving server node, a unique archival policy for each of a plurality of blockchain nodes, executing, by the archiving server node, a consensus mechanism to determine at least one block from a plurality of blocks of the plurality of the blockchain nodes to be archived, and running the unique archival policy to archive the at least one block from the plurality of the blocks.

Data archive

One embodiment provides a method, including: receiving a target unstructured document for determining whether the target unstructured document comprises biased information; identifying an objective of the target unstructured document by extracting, from the target unstructured document, (i) entities and (ii) relationships between the entities; creating a structured knowledge base, wherein the creating comprises (i) creating an entry in the structured knowledge base corresponding to the target unstructured document, (ii) identifying other unstructured documents having a similarity to the target unstructured document, and (iii) generating an entry in the structured knowledge base corresponding to each of the other unstructured documents; applying a bias detection technique on the structured knowledge base; and providing an indication of whether the target unstructured document comprises bias.

Bias detection for unstructured text

A method, computer system, and a computer program product for generating deep learning model insights using provenance data is provided. Embodiments of the present invention may include collecting provenance data. Embodiments of the present invention may include generating model insights based on the collected provenance data. Embodiments of the present invention may include generating a training model based on the generated model insights. Embodiments of the present invention may include reducing the training model size. Embodiments of the present invention may include creating a final trained model.

Deep learning model insights using provenance data

One embodiment provides a method, including: generating a plurality of ontologies wherein each ontology is generated by: monitoring interactions of a user with lineage information, wherein the monitoring comprises monitoring (i) filter interactions and (ii) access interactions; aggregating the monitored interactions of the user with monitored interactions of other users having a given business role; and generating an ontology for the given business role, wherein the subset comprises (i) event types, (ii) event constraints, (iii) event metadata, and (iv) event context; and upon a user having one of the plurality of business roles accessing lineage information on the data platform, providing a subset of the lineage information.

Generation of lineage data subset based upon business role

A method, computer system, and a computer program product for utilizing provenance data to improve machine learning is provided. Embodiments of the present invention may include collecting provenance data. Embodiments of the present invention may include identifying model quality improvements based on the collected provenance data. Embodiments of the present invention may include identifying related models based on the collected provenance data. Embodiments of the present invention may include recommending model quality improvements to a user.

Model quality and related models using provenance data

One embodiment provides a method, including: receiving a plurality of data for job processing, wherein the job processing processes the plurality of data into (i) at least one map phase and (ii) at least one reduce phase; generating a plurality of key-value groups from the plurality of data, wherein the plurality of key-value groups are grouped from data pairs including a key and a value and wherein each of the key-value groups include a grouping of data pairs having a common key and a plurality of values associated with the common key; identifying values common to at least a subset of the key-value groups; generating, based upon the identifying, new key-value groups, wherein at least a subset of the new key-value groups includes key-value groups having common keys and the identified common values; and communicating the new key-value groups to the at least one reduce function for processing.

Sharing intermediate data in map-reduce

In an approach for integrating documents a processor extracts a first set of keywords from at least one structured document. A processor generates a first batch of keywords from the first set of keywords, wherein each keyword in the first batch of keywords includes a weight. A processor extracts a second set of keywords from at least one unstructured document. A processor compares the first batch of keywords to the second set of keywords. A processor determines that the at least one unstructured document matches, based on a predetermined threshold, the at least one structured document, based on the comparison of the first batch of keywords to the second set of keywords. A processor removes the at least one unstructured document from a list of unstructured documents which are to be processed.

System and a method for associating contextual structured data with unstructured documents on map-reduce

Associating contextual structured data with unstructured documents on map-reduce

Methods, systems, and computer program products for automatically suggesting a temporal opportunity for writing one or more sequel articles via artificial intelligence are provided herein. A computer-implemented method includes extracting one or more types of information from a prior written document; automatically determining, based on the extracted information, at least one temporal opportunity for generating a follow-up written document to the prior written document; automatically generating a follow-up written document to the prior written document, the follow-up written document being written in a style that indicates that it is in response to the prior written document, in accordance with the at least one determined temporal opportunity, and based on (i) one or more items of information, related to the extracted information, derived from one or more web sources, and (ii) a writing model attributed to a user.

Automatically suggesting a temporal opportunity for and assisting a writer in writing one or more sequel articles via artificial intelligence

One embodiment provides a method, comprising: receiving a plurality of jobs for processing, wherein each of the plurality of jobs comprises a plurality of tasks and wherein at least one of the plurality of jobs is dependent on another of the plurality of jobs; receiving task dependencies between tasks of the at least one of the plurality of jobs and tasks of the another of the plurality of jobs, wherein the task dependencies identify dependent tasks from the tasks of the at least one of the plurality of jobs and dependee tasks from the tasks of the another of the plurality of jobs; scheduling the processing of the dependent tasks as being based upon only the completed processing of the dependee tasks; and performing job processing of the dependent tasks after processing of the dependee tasks irrespective of the overall job processing status of the another of the plurality of jobs.

Processing jobs using task dependencies

Methods, systems, and computer program products for multi-level colocation and analytical processing of spatial data on MapReduce are provided herein. A method includes correlating multiple items of spatial data and multiple items of attribute data within a file system to generate multiple blocks of correlated data; colocating each of the multiple blocks of correlated data on a given node within the file system based on a data block placement policy; and clustering multiple replicas generated for each of the multiple data blocks at multiple levels of spatial granularity within the file system.

Multi-level colocation and processing of spatial data on MapReduce

Methods and arrangements for managing map-reduce jobs. There are identified intermediate data produced, in a current map-reduce cycle, by a plurality of nodes in the distributed network, the nodes being selected from the group consisting of: a plurality of map nodes, and a plurality of reducer nodes. There are identified a plurality of classes of data, for classifying the intermediate data. Discrete portions of the intermediate data are classified into respective ones of the classes of data, wherein at least one of the classes of data comprises intermediate data which are processed in a map-reduce cycle subsequent to the current map-reduce cycle. Other variants and embodiments are broadly contemplated herein.

Extending a map-reduce framework to improve efficiency of multi-cycle map-reduce jobs

Growing community of inventors

New Delhi, India

Himanshu Gupta