International Business Machines Corporation

Elizabeth T Dettman

Andrew Ronald Freed

Michael W Schroeder

Joel C Dubbels

Michael Terry Payne

Corville O Allen

John Edward Petri

Paul Richard Chmielewski

Jeremy John Salsman

Troy Biesterfeld

Improved data ingestion techniques are provided. A data set comprising records is received, where each record contains one or more fields. A group of fields is identified, where each of the fields has a common metadata attribute. Metrics are determined for the group based on metadata associated with each field, and weight values are assigned to each of the metrics. A natural language processing (NLP) measure and a discreteness measure are generated for the group of fields based on the metrics and the weight values. A processing workflow is selected to use when ingesting data from the group of fields into a corpus, based on comparing the NLP measure and the discreteness measure to one or more predefined thresholds, and each of the fields in the group of fields are processed using the processing workflow.

Inferring confidence and need for natural language processing of input data

Embodiments presented herein disclose techniques for transforming input documents having disparate formats into a normalized format (e.g., Atom, RSS, HTML, customized XML, etc.). According to one embodiment, a plurality of fields is identified in an input document that has a given format. Each field includes a descriptor and text content associated with the descriptor. For each field, semantic properties are evaluated for the descriptor and text content against a plurality of mapping rules to determine whether the field is consistent with one of a plurality of fields of a target format. Each mapping rule specifies characteristics associated with one of the fields in the target format. Once so determined, a mapping from the first field to the second field is defined.

Natural language processing&#x2014;assisted extract, transform, and load techniques

Determining the likelihood that an input descriptor and associated text content match a target field using natural language processing techniques in preparation for an extract, transform and load process

Mechanisms are provided, in a Question and Answer (QA) system comprising a processor and a memory, for debugging code. An input question identifying an error during execution of code is processed by the QA system using a corpus corresponding to a software product associated with the code, thereby generating a first candidate answer set. The QA system processes the input question using a code change record repository identifying changes to the code performed over time to generate a second candidate answer set. The QA system generates a final answer to the input question based on the first and second candidate answer sets and outputs the final answer to the input question. The final answer to the input question identifies at least one of a source, in the code, of the error or a solution to resolving the error.

Debugging code using a question and answer system based on documentation and code change records

An approach is provided in which a knowledge manager analyzes multiple document phrases using a natural language processing model and generates multiple interpretations based upon the analysis. The knowledge manager identifies misinterpretation patterns by comparing the multiple interpretations with multiple corrections that include corrections to the multiple interpretations. In turn, the knowledge manager generates interpretation rules based upon the identified patterns and applies the interpretation rules to the natural language processing model.

Pattern identification and correction of document misinterpretations in a natural language processing system

A mechanism is provided, in a data processing system comprising a processor and a memory configured to implement a question and answer (QA) system, for effectively ingesting data for answering questions in the QA system. A received input question having a set of question characteristics is parsed, which are compared to question characteristics associated with a set of previous questions. Responsive to the set of question characteristics matching the question characteristics associated with one or more previous questions above a related-question predetermined threshold, identification is made as to whether answers to the one or more previous questions were obtained from static information sources or real-time information sources. Responsive to the answers to the one or more previous questions being obtained from the real-time information sources above the predetermined real-time threshold, real-time information sources related to the characteristics of the input question are initially utilized to answer the input question.

Growing community of inventors

Rochester, MN, United States of America

Elizabeth T Dettman