International Business Machines Corporation

Globalfoundries Inc.

Laura Chiticariu

Yunyao Li

Marina Danilevsky Hailpern

Shivakumar Vaithyanathan

Rajasekar Krishnamurthy

Prithviraj Sen

Huaiyu Zhu

Huahai Yang

Alan Akbik

Tanveer Fathima Syeda-Mahmood

Eser Kandogan

Frederick Ralph Reiss

Arnaldo Carreno-Fuentes

Shankar Venkataraman

Daniel Gruhl

Varun Bhagwan

Anbang Xu

Jeffrey Thomas Kreulen

Radu Florian

Hiroshi Kanayama

Robert T Ward

Alexandre Valentinovich Evfimievski

Vitaly Feldman

Siddhartha Brahma

Takuya Ohko

Chenguang Wang

Sudeepa Roy

Bin Liu

Xilun Chen

Efsun Kayi

Victor K Ondego

Yousef El-Kurdi

George A Cypher

Vijaykumar R Bommireddipalli

Richard S Maraschi

Ah-Fung Sit

Aparna Garimella

A computer-implemented method according to one embodiment includes receiving, at a deep neural network (DNN), a plurality of sentences each having an associated label; training the DNN, utilizing the plurality of sentences and associated labels; and producing a linguistic expression (LE) utilizing the trained DNN.

Producing explainable rules via deep learning

An approach for generating synthetic treebanks to be used in training a parser in a production system is provided. A processor receives a request to generate one or more synthetic treebanks from a production system, wherein the request indicates a language for the one or more synthetic treebanks. A processor retrieves at least one corpus of text in which the requested language is present. A processor provides the at least one corpus to a transformer enhanced parser neural network model. A processor generates at least one synthetic treebank associated with a string of text from the at least one corpus of text in which the requested language is present. A processor sends the at least one synthetic treebank to the production system, wherein the production system trains a parser utilized by the production system with the at least one synthetic treebank.

Treebank synthesis for training production parsers

Methods, systems, and computer program products for extracting structure and semantics from tabular data are provided herein. A computer-implemented method includes processing tabular data comprising data cells and header cells, wherein the processing includes: identifying one or more regions within the tabular data, wherein each of the regions comprises one or more of the data cells; matching some of the regions to one or more of the header cells, wherein the matched header cells are semantically related to the data cells inside the matched region; and generating, based on the matching, an output describing semantic relationships between the data cells and the header cells. The method also includes creating, for each data cell, a tuple comprising semantic information contained within one or more of the header cells that pertains to the data cell.

Extracting structure and semantics from tabular data

A computer-implemented method for generating a question from an abstracted template is described. A non-limiting example of the computer-implemented method includes receiving, by a processor, a question. The method parses, by the processor, the question into a parse tree and abstracts, by the processor, an abstracted template from the parse tree. The method receives, by the processor, a domain schema and a domain knowledge base and generates, by the processor, a new question based on the abstracted template, the domain schema, and the domain knowledge base.

Domain-specific labeled question generation for training syntactic parsers

Methods, systems, and computer program products for a semi-automatic process for creating a natural language processing resource are provided herein. A computer-implemented method includes identifying multiple annotation tasks in connection with natural language processing of input text, and automatically determining, based on analysis of (i) parameters related to the identified annotation tasks and (ii) parameters related to annotation task users, routing instructions for the identified annotation tasks, wherein the routing instructions comprise (a) instructions to route a first sub-set of the identified annotation tasks to non-expert annotation task users and (b) instructions to route a second sub-set of the identified annotation tasks to expert annotation task users. The method also includes routing, in accordance with the routing instructions, the first sub-set of the identified annotation tasks to the non-expert annotation task users and the second sub-set of the identified annotation tasks to the expert annotation task users.

Semi-automatic process for creating a natural language processing resource

Methods and arrangements for managing development of information extraction rules. One or more documents are opened for extraction. An interface is provided to create a label and thereupon label a portion of the document. The created label is stored, and an extractor is developed based on the labeling. A test interface is provided for the extractor, and results of a test conducted through the test interface are displayed. The extractor is exported. In accordance with at least one embodiment, developers are presented with eased automated guidance to write extractors, which thereby reduces an overall manual effort involved in extractor development. Generally, a focused, tutorial-type environment serves as a guide based on previously developed best practices.

Building and maintaining information extraction rules

One embodiment provides a method for developing a text analytics program for extracting at least one target concept including: utilizing at least one processor to execute computer code that performs the steps of: initiating a development tool that accepts user input to develop rules for extraction of features of the at least one target concept within a dataset comprising textual information; developing, using the rules for feature extraction, an evaluation dataset comprising at least one document annotated with the at least one target concept to be extracted by the text analytics program; creating, using the rules for feature extraction, a rule-based annotator to extract the at least one target concept; training, using the evaluation dataset, a machine-learning annotator to extract the at least one target concept within the dataset; combining the rule-based annotator and the machine learning annotator to form a combined annotator; evaluating, using the evaluation dataset, extraction performance of the combined annotator against a predetermined threshold; and publishing, when the extraction performance of the combined annotator exceeds the predetermined threshold, the combined annotator for use in an application that extracts the at least one target concept from a plurality of datasets.

Unified text analytics annotator development life cycle combining rule-based and machine learning based techniques

Embodiments relate to facilitating construction of concepts from a task specification. A method includes receiving, from a user via a user interface, a task specification in natural language form. The method also includes parsing the task specification into a plurality of components, and searching a database for an existing concept having a pattern that approximates at least a portion of the plurality of components. The concept includes semantic meanings that are representable by textual patterns. The method further includes identifying any components of the plurality of components that are not included in the existing concept, and building a new concept that combines the existing concept and the components of the plurality of components that are not included in the existing concept.

Constructing concepts from a task specification

One embodiment provides method for constructing a cross-lingual information extraction program, the method including: utilizing at least one processor to execute computer code that performs the steps of: constructing a plurality of language-specific representations from text expressed in a plurality of languages by parsing the text of each language using a language-specific semantic parser; mapping the plurality of language-specific representations to a single cross-lingual semantic representation, wherein the cross-lingual semantic representation encompasses the plurality of languages; and constructing the cross-lingual information extraction program based on the cross-lingual semantic representation. Other aspects are described and claimed.

Cross-lingual information extraction program

One embodiment provides a method for generating a natural language resource using a parallel corpus, the method including: utilizing at least one processor to execute computer code that performs the steps of: receiving, from a parallel corpus, natural language text in a source language and a corresponding translation of the natural language text in a target language, wherein the natural language text in the source language comprises linguistic annotations; projecting the linguistic annotations from the source language natural language text to the target language natural language text; applying one or more filters to remove at least one projected linguistic annotation from the target language natural language text that results in at least one error; selecting at least one target language natural language text having substantially complete linguistic annotations; training a machine learning model using the selected at least one target language natural language text and annotations; and adding, using the trained machine learning model, linguistic annotations to at least one target language natural language text having incomplete linguistic annotations. Other aspects are described and claimed.

Generation of a natural language resource using a parallel corpus

A method for extracting information from electronic documents, including: learning terms and term variants from a training corpus, wherein the terms and the term variants correspond to a specialized dictionary related to the training corpus; generating a list of negative indicators found in the training corpus; performing a partial match of the terms and the term variants in a set of electronic documents to create initial match results; and performing a negation test using the negative indicators and a positive terms test using the terms and the term variants on the initial match results to remove matches from the initial match results that fail either the negation test or the positive terms test, resulting in final match results.

Extraction of information from clinical reports

Probabilistic surfacing of potentially sensitive identifiers is provided. In one embodiment of the present invention, a method of and computer program product for surfacing of potentially sensitive identifiers are provided. An input string is read. The input string has a length. The input string is divided into a plurality of tokens. Each of the tokens has a predetermined length. A score is determined for each of the plurality of tokens. A composite score is determined based on the scores of each of the plurality of tokens. Whether the input string comprises an identifier is determined by comparing the composite score to a predetermined threshold.

Probabilistic surfacing of potentially sensitive identifiers

A computer identifies and ranks URL hyperlinks to possible pirated media content by searching a web page from a first website for one or more indicator keywords, wherein a strength of an indicator keyword is related to a likelihood of pirated media content. Responsive to locating a plurality of instances of the one or more indicator keywords, identifying a plurality of hyperlinks respectively associated with one or more of the plurality of instances. Weighting, the identified plurality of hyperlinks based on at least one of: a strength of associated indicator keywords, number of associated indicator keywords, number of times each hyperlink was identified, and date of posting. Ranking the plurality of hyperlinks according to weight indicating a relative likelihood that respective hyperlinks point to pirated media content in a ranked list.

Identifying and ranking pirated media content

A phrase matching system is described. The system includes a training engine and a matching engine. The training engine is configured to: learn terms and term variants from a training corpus, wherein the terms and the term variants correspond to a specialized dictionary related to the training corpus; and generate a list of negative indicators found in the training corpus. The matching engine is configured to: perform a partial match of the terms and the term variants in a set of electronic documents to create initial match results; and perform a negation test using the negative indicators and a positive terms test using the terms and the term variants on the initial match results to remove matches from the initial match results that fail either the negation test or the positive terms test, resulting in final match results.

Growing community of inventors

San Jose, CA, United States of America

Laura Chiticariu