International Business Machines Corporation

Ritwik Ray

Scott Robert Carrier

Hui Wang

Jonathan Rand

Jothilakshmi Sirangimoorthy

Robert Fredenburg

Marie Angelopoulos

Maria Gabrani

Frederick Roberts

Christopher Gagen

Provided are a computer program product, system, and method for pre-processing a table in a document for natural language processing (NLP). A graphical user interface (GUI) provides a representation of table items in a table in a document including a set of a main element comprising an entity whose value is to be extracted, a conditional element that refines the entity, and a value element comprising a value for the entity. Graphical controls are rendered in the GUI to enable a user to select an element from the table to be the main element, conditional element, and value element. The set of the main element, conditional element, and value element are updated with the user selected element to form a modified set. The modified set of the main element, conditional element, and the value element are provided to an NLP engine to perform natural language processing.

Pre-processing a table in a document for natural language processing

Provided are a computer program product, system, and method for pre-processing a table in a document for natural language processing. A table in a document is parsed to extract column headers, row headers, and data cells, which are processed to determine an initial set of a main element comprising an entity whose value is to be extracted, a conditional element that refines the entity, and a value element comprising a value for the entity. A user selection is received of at least one of the column headers, row headers, and data cells for at least one of the main element, conditional element, and the value element in the initial set to produce a modified set of the main element, conditional element, and value element. The modified set is provided to a natural language processing engine to perform natural language processing of the document including the table, using the modified set.

Aspects of the present disclosure describe techniques for generating a machine learning model for extracting information from textual content. The method generally includes receiving a training data set including a plurality of documents having related textual strings. A relevancy model is generated from the training data set. The relevancy model is generally configured to generate relevance scores for a plurality of words extracted from the plurality of documents. A knowledge graph model illustrating relationships between the plurality of words extracted from the plurality of documents is generated from the training data set. The relevancy model and the knowledge graph model are aggregated into a complimentary model including a plurality of nodes from the knowledge graph model and weights associated with edges between connected nodes, wherein the weights comprise relevance scores generated from the relevancy model, and the complimentary model is deployed for use in analyzing documents.

Extracting information from unstructured documents using natural language processing and conversion of unstructured documents into structured documents

Aspects of the present disclosure describe techniques for generating a machine learning model for extracting information from textual content. The method generally includes receiving an unstructured document and a structured document including information extracted from the unstructured document and position information associated with the extracted information. The unstructured document is rendered in a first pane, and a graphical rendering of the structured document is rendered in a second pane. The graphical rendering generally may be a structure in which content from the structured document is displayed in a hierarchical format. Each element in the structured document is linked to the rendered unstructured document based on position information included in the structured document.

Navigating unstructured documents using structured documents including information extracted from unstructured documents

Methods and systems are provided to extract information within complex documents, and the extracted information may be compared to identify differences between complex documents or the extracted information may be analyzed with respect to the individual document. Information is extracted from complex documents comprising unstructured data to create a structured data repository, or analytics knowledge base. This database may be utilized to compare concepts that are common to one or more documents, allowing ease of comparison of documents, and identification of information that is different or identification of (same or similar) information that is presented differently in a set of complex documents.

Growing community of inventors

Apex, NC, United States of America

Ritwik Ray