Sign language recognition using parameter keypoints

Abstract

A system captures image data that includes a person executing a sign of a sign language. Keypoints are extracted from the image data. The keypoints include locations on a hand of the person, an arm of the person, a trunk of the person and a face of the person. The keypoints are provided to a parameter analyzer for detecting characteristics of the sign. The characteristics of the sign includes a handshape, a palm orientation, an articulation point and a movement. A description of the sign is generated from an output of the parameter analyzer. The description of the sign is filtered and aggregated, thereby creating a condensed description of the sign. A database of known signs is queried with the condensed description using a dynamic time warping algorithm. A word is retrieved from the database that is associated with the sign.