System and method for video instance segmentation via multi-scale spatio-temporal split attention transformer

Abstract

A system, method, computer readable storage medium for a computer vision system includes at least one video camera, and video processor circuitry. The method includes inputting a stream of video data and generating a sequence of image frames, segmenting and tracking, by the video analysis apparatus, object instances in the stream of video data, including receiving the sequence of image frames, analyzing the sequence of image frames using a video instance segmentation transformer to obtain a video instance mask sequence from the sequence of image frames, the transformer having a backbone network, a transformer encoder-decoder, and an instance matching and segmentation block, The encoder contains a multi-scale spatio-temporal split attention module to capture spatio-temporal feature relationships at multiple scales across multiple frames. The decoder contains a temporal attention block for enhancing a temporal consistency of transformer queries. The method includes displaying the video instance mask sequence.