Skeleton-based action detection using recurrent neural network

Abstract

In implementations of the subject matter described herein, an action detection scheme using a recurrent neural network (RNN) is proposed. Joint locations for a skeleton representation of an observed entity in a frame of a video and a predefined action label for the frame are obtained to train a learning network including RNN elements and a classification element. Specifically, first weights for mapping the joint locations to a first feature for the frame generated by a first RNN element in a learning network and second weights for mapping the joint locations to a second feature for the frame generated by a second RNN element in the learning network are determined based on the joint locations and the predefined action label. The first and second weights are determined by increasing a first correlation between the first feature and a first subset of the joint locations and a second correlation between the second feature and the first subset of the joint locations. Based on the joint locations and the predefined action label, a parameter for a classification element included in the learning network is also determined by increasing a probability of the frame being associated with the predefined action label. The probability is generated by the classification element at least based on the first and second features.