System and method of bridging the gap between object and image-level representations for open-vocabulary detection

Abstract

An object detection system and method in which a machine learning engine is configured with a region-based knowledge distillation stage that generates region embeddings from a training image having bounding boxes. A linear layer learns a region-level vision-language mapping for projecting feature embeddings from the training image to a common feature space shared by text embeddings to obtain the region embeddings. An image-level supervision stage generates pseudo-box labels for a classification training image and region embeddings from the training image having bounding boxes and corresponding class labels and the classification training image having an image-level label as input. Pseudo-box labels are determined on the classification training image as an image-level vision-language mapping. A weight transfer function conditions the image-level vision-language mapping on the learned region-level vision-language mapping. A trained object detector outputs a newly captured image annotated with a bounding box for a novel object.