Generating text prompts for digital images utilizing vision-language models and contextual prompt learning

Abstract

The present invention relates to systems, methods, and non-transitory computer-readable media that implements a vision language machine learning model to generate text representations of digital image from localized context tokens, where the text representations semantically reflect the content of the digital image under consideration. In particular, in some embodiments, the disclosed invention generates image patch feature representations that represent patches from an input image. Further, in some embodiments, the disclosed invention generates localized context tokens from the image patch feature representations and prompt context tokens. Moreover, in some embodiments, by utilizing the localized context tokens, the disclosed invention generates a text representation by utilizing a text encoder of the vision language machine learning model.