System and method for contrastive affinity learning via auxiliary prompts for generalized novel category discovery

Abstract

A system and method of fine-grained image category discovery with few human annotations includes a camera and a trained machine learning model, which predicts a label for an object in a captured image and outputs the predicted label. The machine learning model is trained by contrastive affinity learning, including retrieving images having an object, a warm-up stage in which semi-supervised contrastive learning is performed based on projected features of a class token and an ensembled prompt, respectively. In a contrastive affinity learning stage, a student model and an exponentially moving averaged teacher model are forwarded with different augmented views of the retrieved images. Teacher embeddings are enqueued into a token-specific memory. A semi-supervised contrastive loss is computed on a current batch and a contrastive affinity learning loss for student embeddings and the teacher embeddings with pseudo-labels from a affinity graph dynamically generated by semi-supervised affinity generation.