Adaptive human instance segmentation with stereo view consistency

Abstract

A system stores first and second images generated by first and second cameras; applies a segmentation model to the first image to generate a first segmentation mask identifying object instances; applies the segmentation model to the second image to generate a second segmentation mask identifying the object instances; projects the first segmentation mask to a viewpoint of the second camera to generate a first projected segmentation mask; converts the first projected segmentation mask and the second segmentation mask to first and second semantic masks, respectively; and computes a first similarity value based on the first and second semantic masks. This may be repeated exchanging the first and second images to compute a second similarity value. The system determines a loss value based on the first similarity value and the second similarity value and trains the segmentation model based on the loss value.