System and method with visual concept framework for self-supervised semantic segmentation

Abstract

A computer-implemented system and method includes generating first pseudo segment data from a first augmented image and generating second pseudo segment data from a second augmented image. The first augmented image and the second augmented image are in a dataset along with other augmented images. A machine learning system is configured to generate pixel embeddings based on the dataset. The first pseudo segment data and the second pseudo segment data are used to identify a first set of segments that a given pixel belongs with respect to the first augmented image and the second augmented image. A second set of segments is identified across the dataset. The second set of segments do not include the given pixel. A local segmentation loss is computed for the given pixel based on the corresponding pixel embedding that involves attracting the first set of segments while repelling the second set of segments. The global concept loss is computed based on a similarity determination between the first set of segments and a concept vector of a corresponding concept. The corresponding concept categorizes the first set of segments with other sets of segments across the dataset based on semantic meaning. The parameters of machine learning system are updated based on the total loss that takes into account at least the local segmentation loss and the global concept loss.