Disney Enterprises, Inc.

Michail Raptis

Leonid Sigal

Tian Lan

Nataliya Shapovalova

Methods and systems for video action recognition using poselet keyframes are disclosed. An action recognition model may be implemented to spatially and temporally model discriminative action components as a set of discriminative keyframes. One method of action recognition may include the operations of selecting a plurality of poselets that are components of an action, encoding each of a plurality of video frames as a summary of the detection confidence of each of the plurality of poselets for the video frame, and encoding correlations between poselets in the encoded video frames.

Methods and systems for action recognition using poselet keyframes

The disclosure provides an approach for detecting and localizing action in video. In one embodiment, an action detection application receives training video sequences and associated eye gaze fixation data collected from a sample of human viewers. Using the training video sequences and eye gaze data, the action detection application learns a model which includes a latent regions potential term that measures the compatibility of latent spatio-temporal regions with the model, as well as a context potential term that accounts for contextual information that is not directly produced by the appearance and motion of the actor. The action detection application may train this model in, e.g., the latent structural SVM framework by minimizing a cost function which encodes the cost of an incorrect action label prediction and a mislocalization of the eye gaze. During training and thereafter, inferences using the model may be made using an efficient dynamic programming algorithm.

Eye gaze driven spatio-temporal action localization

The disclosure provides an approach for detecting objects in images. An object detection application receives a set of training images with object annotations. Given these training images, the object detection application generates semantic labeling for object detections, where the labeling includes lower-level subcategories and higher-level visual composites. In one embodiment, the object detection application identifies subcategories using an exemplar support vector machine (SVM) based clustering approach. Identified subcategories are used to initialize mixture components in mixture models which the object detection application trains in a latent SVM framework, thereby learning a number of subcategory classifiers that produce, for any given image, a set of candidate windows and associated subcategory labels. In addition, the object detection application learns a structured model for object detection that captures interactions among object subcategories and identifies discriminative visual composites, using subcategory labels and spatial relationships between subcategory labels to reason about object interactions.

Growing community of inventors

Washington, DC, United States of America

Michail Raptis