International Business Machines Corporation

Shanmukha Chaitanya Guttula

Hima Patel

Pranay Kumar Lohia

Nitin Kumar Gupta

Akshar Kaul

Aswin Kannan

Balaji Ganesan

Methods, systems, and computer program products for automatically improving data annotations by processing annotation properties and user feedback are provided herein. A computer-implemented method includes obtaining data annotation pairs, each comprising an input data annotation in a first format and a corresponding output data annotation in a second format; determining, within at least a portion of the data annotation pairs, one or more non-diffs; identifying, across the at least a portion of data annotation pairs, data annotation properties associated with multiple intents by processing the non-diffs using property-related rules; modifying at least a portion of the data annotation pairs based on the identified data annotation properties; outputting the modified data annotation pairs to at least one user; and generating a final collection of data annotation pairs by processing at least a portion of the modified data annotation pairs and user feedback received in response to the outputting.

Automatically improving data annotations by processing annotation properties and user feedback

Methods, systems, and computer program products for classifying data from de-identified content are provided herein. A computer-implemented method includes applying one or more rules to identify one or more structural elements of a document; determining, based at least in part on the one or more structural elements, one or more pairs of words within the document having a hypernym relationship; extracting de-identified content within the document based on one or more de-identification techniques applied to the document; and applying a set of causal rules to the de-identified content and the one or more pairs of words to annotate at a least a portion of the de-identified content as belonging to a class of protected content.

Classifying data from de-identified content

Selecting and ordering the execution of data quality rules includes generating a snapshot of a table-formatted dataset. The snapshot comprises a reduced number of rows of the dataset such that each column variation of the dataset is included in the snapshot. A predetermined collection of data quality (DQ) rules is executed on the snapshot. One or more performance statistics is determined for each of the DQ rules. The performance statistics indicate a likelihood that a DQ rule determines a data quality deficiency. Based on the performance statistics, a subset of the DQ rules is generated. Each DQ rule of the subset is selected based on the likelihood that the DQ rule selected detects a quality deficiency. An ordered subset of selected DQ rules is generated by ordering the application of each of the subset of DQ rules selected. The ordering specifies a sequence for executing each selected DQ rule.

Automated selection and ordering of data quality rules during data ingestion

A computer implemented method transforms data. Responsive to receiving a data transformation of an input string to an output string, a computer system identifies mappable tokens in the input string that are mappable to the output string. The computer system creates a set of initial mappings for a set of common tokens in the mappable tokens. The set of initial mappings maps the set of common tokens from the input string to the output string. The computer system creates a set of user mappings that maps the mappable tokens from input string to the output string using a user input to the set of initial mappings. The computer system generates program code that transform input strings to output strings using the set of user mappings that maps the mappable tokens from input string to the output string, wherein the program code is used to transform input strings to output strings.

Growing community of inventors

Vijayawada, India

Shanmukha Chaitanya Guttula