Cycle-consistent refinement of prompts provided to models

Abstract

This disclosure presents a technique for enhancing the performance of models such as large multimodal models (LMMs) without retraining or fine-tuning. This technique includes an iterative refinement process implemented by three main components: a forward generator, a backward generator, and a discriminator. The forward generator translates a prompt into an output in a different modality, the backward generator translates this output back into the original modality, and the discriminator compares the prompt and the translated prompt to generate a hint for refining the prompt to reduce differences. This cycle continues until the original prompt and the translated prompt match, achieving cycle consistency. The solution offers several advantages, including improving model performance without the need for costly fine-tuning, training data, or expertise. It simplifies system complexity by not relying on external environments like compilers and APIs and uses cycle consistency as a supervisory signal to iteratively refine a prompt.