Multi-objective dense open-vocabulary image recording

Abstract

A multi-objective dense open-vocabulary system includes an image encoder and a classifier. The image encoder includes a summarization contrastive language image pre-training (CLIP) head trained on supervised losses from unlabeled and labeled image data. The summarization CLIP head loses open-vocabulary capabilities as capacity grows, and offsets the loss with pseudo-labels generated by a dense CLIP head. The summarization CLIP head is operational to receive captured images from a source device, and generate image embeddings based on current images. The classifier is operational to receive one or more targets from a text encoder, receive the plurality of image embeddings from the summarization CLIP, classify the plurality of image embeddings to identify one or more output images that contain the one or more targets, and present the one or more output images to the destination device.