Confidence calibration using pseudo-accuracy

Abstract

Systems and methods for training machine learning models are disclosed. An example method includes receiving a plurality of first outputs and a ground truth value for each first output, each first output including an extracted string and a raw confidence score, determining, for each first output, an accuracy metric based at least in part on the extracted string and its corresponding ground truth value, for each extracted string: determining a similarity metric between the respective extracted string and each other extracted string of the plurality of first outputs, and determining a pseudo-accuracy based at least in part on the determined similarity metrics and the determined accuracy metrics, generating training data based at least in part on the determined pseudo-accuracies and the plurality of first outputs, and training the machine learning model, based on the training data, to predict pseudo-accuracies associated with subsequent outputs from a document extraction model.