Microsoft Technology Licensing, LLC

Junliang Xing

Cuiling Lan

Wenjun Zeng

Wentao Zhu

Sijie Song

Yanghao Li

Various implementations of the subject matter described herein relate to a neural network-based action detection. There is provided an action detection scheme using a neural network. The action detection scheme can design and optimize the neural network model based on respective importance of different frames such that frames that are more important or discriminative for action recognition tend to be assigned with higher weights and frames that are less important or discriminative for action recognition tend to be assigned with lower weights.

Neural network-based action detection

In implementations of the subject matter described herein, an action detection scheme using a recurrent neural network (RNN) is proposed. Representation information of an incoming frame of a video and a predefined action label for the frame are obtained to train a learning network including RNN elements and a classification element. The representation information represents an observed entity in the frame. Specifically, parameters for the RNN elements are determined based on the representation information and the predefined action label. With the determined parameters, the RNN elements are caused to extract features for the frame based on the representation information and features for a preceding frame. Parameters for the classification element are determined based on the extracted features and the predefined action label. The classification element with the determined parameters generates a probability of the frame being associated with the predefined action label. The parameters for the RNN elements are updated according to the probability.

On-line action detection using recurrent neural network

In implementations of the subject matter described herein, an action detection scheme using a recurrent neural network (RNN) is proposed. Joint locations for a skeleton representation of an observed entity in a frame of a video and a predefined action label for the frame are obtained to train a learning network including RNN elements and a classification element. Specifically, first weights for mapping the joint locations to a first feature for the frame generated by a first RNN element in a learning network and second weights for mapping the joint locations to a second feature for the frame generated by a second RNN element in the learning network are determined based on the joint locations and the predefined action label. The first and second weights are determined by increasing a first correlation between the first feature and a first subset of the joint locations and a second correlation between the second feature and the first subset of the joint locations. Based on the joint locations and the predefined action label, a parameter for a classification element included in the learning network is also determined by increasing a probability of the frame being associated with the predefined action label. The probability is generated by the classification element at least based on the first and second features.

Growing community of inventors

Beijing, China

Junliang Xing