Mining unlabeled images with vision and language models for improving object detection

Abstract

A method for object detection obtains, from a set of RGB images lacking annotations, a set of regions that include potential objects, a bounding box, and an objectness score indicating a region prediction confidence. The method obtains, by a region scorer for each region in the set, a category from a fixed set of categories and a confidence for the category responsive to the objectness score. The method duplicates each region in the set to obtain a first and a second patch. The method encodes the patches to obtain an image vector. The method encodes a template sentence using the category to obtain a text vector for each category. The method compares the image vector to the text vector via a similarity function to obtain a similarity probability based on the confidence. The method defines a final set of pseudo labels based on the similarity probability being above a threshold.