Systems and methods for labeling training data for information extraction systems

Abstract

A system for extracting a number of data elements from one or more data sources. The system increases the size of training examples that can be used to test, score, and generate the extraction procedure by generating additional training examples. The additional training examples are generated by automatically labeling unlabeled examples and augmented the labeled training examples with the unlabeled examples for which a ground truth value has been estimated. The system queries a number of language models to extract the information from the unlabeled examples and uses an algorithm to estimate the ground truth value from the values estimated by the ensemble of language models. A flag is also generated indicating those unlabeled examples of particular difficulty which may have high uncertainty and require supplemental validation of the estimated ground truth value. The system can populate an ontological data store using the extraction procedure developed using the additional training examples.