Adaptive hand tracking and gesture recognition using face-shoulder feature coordinate transforms

Abstract

A method for recognizing hand gestures when the hand is held in front of the user's upper body. The method detects dominant features associated with the upper body, including features on the face (eyes and mouth etc.) and the shoulder extremes. Geometrical relations among these features are used to judge whether the face/head has turned to the side relative to the shoulder. If not, a face-shoulder coordinate system is established using these face-shoulder features, and the hand images are transformed into this new coordinate system. This in effect defines hand gestures relative to the user's face and shoulder, thus significantly reducing the dimensions of the feature space for hand gestures. This face-shoulder coordinate transform can be combined either with traditional appearance-based hand gesture recognition methods, or with a conventional or hybrid deep neural network for RGB or RGB-D images.