Direct preference optimization data filtering via answer similarity using language model embeddings

Abstract

Disclosed herein are system, method, apparatus, and computer-readable medium embodiments for filtering data for direct preference optimization training via answer similarity using language model embeddings. A system inputs an accepted question-response pair and a rejected question-response pair into a language model. The system extracts a first embedding for the accepted question-response pair and a second embedding for the rejected question-response pair from the language model. Using the first embedding and the second embedding, the system further computes a similarity score for the accepted question-response pair and the rejected question-response pair. Finally, based on the similarity score satisfying a threshold, the system includes the rejected question-response pair in a dataset and trains the language model using the dataset.