Temporally coherent segmentation of rgbt volumes with aid of noisy or incomplete auxiliary data

Abstract

An apparatus for performing video segmentation, including image sensors of a first type, auxiliary sensors of a second type, and a processor executing computer-executable instructions stored in memory. The instructions include capturing RGB image video data from a plurality of first image sensors; capturing second video data from a plurality of second, auxiliary sensors; for a reference frame, determining an initial segmentation map by segmenting a first object-of-interest from a background from said second video data; determining a history of segmentation information on a pixel-by-pixel basis, wherein a length of said history is based on motion data for the object-of-interest; generating a refined segmentation map by refining on a pixel-by-pixel basis a corresponding value of said initial segmentation map based on said history of segmentation information; and performing RGB segmentation on a corresponding reference frame of said RGB image video based on said refined segmentation map.