Weakly-supervised referring expression segmentation

Abstract

The present disclosure relates to systems, methods, and non-transitory computer-readable media that modifies parameters of a fused feature extractor. In particular, the disclosed systems generate inferred masks from digital images and digital text prompts using a fused feature extractor. Furthermore, the disclosed systems identify a subset of the inferred masks that satisfy a validity threshold. Moreover, the disclosed systems generate an augmented training set by combining the subset of the inferred masks with a training set that includes the ground truth masks. Further, the disclosed systems generate object mask predictions from the augmented training set and determine ground truth and pseudo measures of loss by comparing the object mask predictions with the inferred masks and the ground truth masks. From the ground truth and pseudo measures of loss, the disclosed systems modify parameters of the fused feature extractors.