Machine-learned model alignment with synthetic data

Abstract

Aspects of the disclosed technology include computer-implemented systems and methods for adapting machine-learned models using high-quality synthetic data that is tailored to elicit improved instruction-following abilities for particular target instruction distributions and models. A model adaptation system can obtain instruction metadata indicative of at least one use case and at least one skill associated with a particular computing task to be performed by a target machine-learned model. The system can generate a metadata-conditioned synthetic instruction by prompting a generative model system including one or more machine-learned generative models with the instruction metadata as one or more constraints. The system can generate a model response by prompting the generative model system with the metadata-conditioned synthetic instruction. The system can modify a target sequence processing model based at least in part on a data pair including the metadata-conditioned synthetic instruction and the model response.