Language machine learning model for image recognition tasks

Abstract

A computer-readable storage medium and method for adapting a vision-language (VL) machine learning model from a pre-trained VL model. The pre-trained VL model includes a pre-trained image encoder and a pre-trained textual encoder. The method is applicable to few shot learning and training of learnable prompts. After feature extraction, a machine learning engine determines a cross-entropy loss from prompted image features and prompted textual features. The machine learning engine then determines a self-regularizing consistency loss from the prompted image features, pre-trained image features, the prompted textual features, and pre-trained textual features. The learnable text and image prompts are updated based on the cross-entropy loss in combination with the self-regularizing consistency loss. To preserve generalization for downstream tasks, the updating with the self-regularizing consistency loss is constrained to maximize mutual agreement between the prompted image and textual features and the pre-trained image and textual features.