Scalable prompt learning for large vision-language models

Abstract

A method of generating text-driven prompts and class prediction probabilities using a vision-language model (VLM) includes receiving candidate class names associated with a plurality of candidate classes for images, generating class text tokens based on a text description of the candidate class names, and generating a plurality of context prompt vectors using a prompt generator. The context prompt vectors define context information associated with an image classification task to be performed by the VLM. The method further includes generating prompts for each of the plurality of candidate classes by appending respective class text tokens to the context prompt vectors for each of the plurality of candidate classes, and, using the VLM, generating and outputting a class prediction probability for a sample image based on the plurality of context prompt vectors.