Facebook Technologies, LLC

Shangchen Han

Robert Y Wang

Christopher David Twigg

Yuting Ye

Po-Chen Wu

A system tracks poses of a passive object using fiducial markers on fiducial surfaces of a polygonal structure of the object using image data captured by a camera. The system includes an object tracking controller that generates an estimated pose for a frame of the image data using an approximate pose estimation (APE), and then updates the estimated pose using a dense pose refinement (DPR) of pixels. The APE may include minimizing reprojection error between projected image points of the fiducial markers and observed image points of the fiducial markers in the frame. The DPR may include minimizing appearance error between image pixels of the fiducial markers in the frame and projected model pixels of the fiducial markers determined from the estimated pose and the object model. In some embodiments, an inter-frame corner tracking (ICT) of the fiducial markers may be used to facilitate the APE.

Passive object tracking using camera

A system tracks a user's hands by processing image data captured using one or more passive cameras. The system includes one or more passive cameras, such as color or monochrome cameras, and a hand tracking unit. The hand tracking unit receives the image data of the user's hand from the one or more passive cameras. The hand tracking unit determines, based on applying the image data to a neural network model, heat maps indicating locations of hand features of a hand shape model. The hand tracking unit may include circuitry that implements the neural network model. The neural network model is trained using image data from passive cameras, depth cameras, or both. The hand tracking unit determines a hand pose of the user's hand based on the locations of the hand features of the hand shape model. The hand pose may be used as a user input, or to render the hand for a display, such as in a head-mounted display.

Hand tracking using a passive camera system

A tracking system converts images to a set of points in 3D space. The images are of a wearable item that includes markers, and the set of points include representations of the markers. A view is selected from a plurality of views using the set of points, and the selected view includes one or more representations of the representations. A depth map is generated based on the selected view and the set of points, and the depth map includes the one or more representations. A neural network maps labels to the one or more representations in the depth map using a model of a portion of a body that wears the wearable item. A joint parameter is determined using the mapped labels. The model is updated with the joint parameter, and content provided to a user of the wearable item is based in part on the updated model.

Growing community of inventors

Kirkland, WA, United States of America

Shangchen Han