Training checkpoint validation for machine learning models

Abstract

Systems and methods are provided for a prompt and content generation service to validate checkpoints of large language models (LLMs). The prompt and content generation service may execute use case scenarios against checkpoints of LLMs using a prompt and expected output regarding each of the use case scenarios. The prompt and content generation service may then retrieve or receive generated outputs by the LLMs to compare against the expected outputs for each use case scenario. By comparing the expected outputs to the generated outputs, the prompt and content generation service may determine a degree of matching for each use scenario and an overall score taking into the account the degree of matching for all of the use case scenarios.