Methods of real-time spatio-temporal activity detection and categorization from untrimmed video segments

Abstract

Methods of detecting and categorizing an action in an untrimmed video segment regardless of the scale of the action and the close proximity of other actions. The methods improve upon the prior art which either require trimmed video segments including only a single activity depicted therein, or untrimmed video segments including relatively few actions, persons, or objects of interest, thereby directing the classification. Instead, the methods utilize a plurality of tubelets used to represent discreet actions, persons, and objects of interest within the comprehensive untrimmed video segment. The tubelets are localized to correct for pixel-level foreground-background biases, which are then turned into short spatio-temporal action tubelets that are passed to a classification network to obtain multi-label predictions. After classification, the tubelets are be linked together to obtain the final detections with varying lengths, and the method merges the short action tubelets into final action detections.