Methods, systems, and media for identifying human coactivity in images and videos using neural networks

Abstract

Methods, systems and processor-readable media for classifying human coactivity performed jointly by two humans shown in an image or a sequence of frames of a video. A 2D convolutional neural network is used to identify key points on the human body, such as human body joints, visible within the image or within each frame, for each of the two people performing the coactivity. An encoded representation of the key points is created for each image or frame, the encoded representation being based on distances between the key points of the first person and key points of the second person. The encoded representation for the image, or a concatenated volume of the encoded representations of the frames, is processed by a fully-connected neural network trained to classify the coactivity.