Generating segmentation masks for objects in digital videos using pose tracking data

Abstract

The present disclosure relates to systems, methods, and non-transitory computer-readable media that generate joint-based segmentation masks for digital objects portrayed in digital videos. In particular, in one or more embodiments, the disclosed systems utilize a video masking model having a pose tracking neural network and a segmentation neural network to generate the joint-based segmentation masks. To illustrate, in some embodiments, the disclosed systems utilize the pose tracking neural network to identify a set of joints of the digital object across the frames of the digital video. The disclosed systems further utilize the segmentation neural network to generate joint-based segmentation masks for the video frames that portray the object using the identified joints. In some cases, the segmentation neural network includes a multi-layer perceptron mixer layer for mixing visual features propagated via convolutional layers.