Facebook Technologies, LLC

Yuting Ye

Robert Y Wang

Christopher David Twigg

Shangchen Han

Po-Chen Wu

A system generates a user hand shape model from a single depth camera. The system includes the single depth camera and a hand tracking unit. The single depth camera generates single depth image data of a user's hand. The hand tracking unit applies the single depth image data to a neural network model to generate heat maps indicating locations of hand features. The locations of hand features are used to generate a user hand shape model customized to the size and shape of the user's hand. The user hand shape model is defined by a set of principal component hand shapes defining a hand shape variation space. The limited number of principal component hand shape models reduces determination of user hand shape to a smaller number of variables, and thus provides for a fast calibration of the user hand shape model.

Hand calibration using single depth camera

A system generates a user hand shape model from a single depth camera. The system includes the single depth camera and a hand tracking unit. The single depth camera generates single depth image data of a user's hand. The hand tracking unit applies the single depth image data to a neural network model to generate heat maps indicating locations of hand features. The locations of hand features are used to generate a user hand shape model customized to the size and shape of the user's hand. The user hand shape model is defined by a set of principle component hand shapes defining a hand shape variation space. The limited number of principle component hand shape models reduces determination of user hand shape to a smaller number of variables, and thus provides for a fast calibration of the user hand shape model.

A system tracks poses of a passive object using fiducial markers on fiducial surfaces of a polygonal structure of the object using image data captured by a camera. The system includes an object tracking controller that generates an estimated pose for a frame of the image data using an approximate pose estimation (APE), and then updates the estimated pose using a dense pose refinement (DPR) of pixels. The APE may include minimizing reprojection error between projected image points of the fiducial markers and observed image points of the fiducial markers in the frame. The DPR may include minimizing appearance error between image pixels of the fiducial markers in the frame and projected model pixels of the fiducial markers determined from the estimated pose and the object model. In some embodiments, an inter-frame corner tracking (ICT) of the fiducial markers may be used to facilitate the APE.

Passive object tracking using camera

A system tracks a user's hands by processing image data captured using one or more passive cameras. The system includes one or more passive cameras, such as color or monochrome cameras, and a hand tracking unit. The hand tracking unit receives the image data of the user's hand from the one or more passive cameras. The hand tracking unit determines, based on applying the image data to a neural network model, heat maps indicating locations of hand features of a hand shape model. The hand tracking unit may include circuitry that implements the neural network model. The neural network model is trained using image data from passive cameras, depth cameras, or both. The hand tracking unit determines a hand pose of the user's hand based on the locations of the hand features of the hand shape model. The hand pose may be used as a user input, or to render the hand for a display, such as in a head-mounted display.

Growing community of inventors

Redmond, WA, United States of America

Yuting Ye