Active sparse labeling of video frames

Abstract

An active sparse labeling system that provides high performance and low annotation costs by performing partial instance annotation (i.e., sparse labeling) by frame level selection to annotate the most informative frames, thereby improving action detection task efficiencies. The active sparse labeling system utilizes a frame level cost estimation to determine the utility of each frame in a video based on the frame's impact on action detection. The system includes an adaptive proximity-aware uncertainty model, which is an uncertainty-based frame scoring mechanism. The adaptive proximity-aware uncertainty model estimates a frame's utility using the uncertainty of detections of the frame's proximity to existing annotations, thereby determining a diverse set of frames in a video which are effective for learning the task of dense video understanding (such as action detection). In addition, the active sparse labeling system includes a loss formulation training model (max-Gaussian weighted loss) that uses weighted pseudo-labeling.