System and method for generating high quality labeled extraction datasets for training machine learning models

Abstract

A labeled extraction dataset for training machine learning models for information extraction tasks is generated using an ensemble of machine learning models serving as annotators. The plurality of annotators generate a plurality of labels and associated confidences for each field in an input data source. The plurality of labels for each field are consolidated to generate a single label and associated confidence per field of the input data source. The quality of the resulting label is scored to identify high quality and low quality labels. The high quality labels are stored in a final labeled extraction dataset and the low quality labels are provided to human labels for review and correction. The final labeled extraction dataset may then be used in training machine learning models for information extraction tasks such as document understanding.