Leveraging a large language model encoder to evaluate predictive models

Abstract

Aspects of the disclosure include methods for evaluating a predictive model. An exemplary method includes training an evaluation model to output, for an input first entity-second entity pair, a content relevancy prediction. A large language model encoder of the evaluation model generates a first embedding for the first entity and a second embedding for the second entity. The embeddings are fed to an interaction tower to produce a logit and the logit is passed with true labels to a loss function for fine-tuning. The true labels include labeled training data generated by modifying training data having a first proportion of negative labeled data to provide a second proportion of negative labeled data greater than the first proportion. The evaluation model is used to score a performance of a predictive model based at least in part on a comparison of predictions made by the respective models for a same entity pair.