Facebook Technologies, LLC

Dawei Wang

Mohsen Shahmohammadi

Jason Saragih

Shih-En Wei

Hernan Badino

Michal Perdoch

Alexander Trenor Hypes

Yaser Sheikh

James White

Ying Yang

A virtual reality (VR) or augmented reality (AR) head mounted display (HMD) includes multiple image capture devices positioned within the HMD to capture portions of a face of a user wearing the HMD. Images from an image capture device include a user's eye, while additional images from another image capture device include the user's other eye. The images and the additional images are provided to a controller, which applies a trained model to the images and the additional images to generate a vector identifying a position of the user's head and positions of the user's eye and fixation of each of the user's eyes. Additionally, illumination sources illuminating portions of the user's face include in the images and in the additional images are configured when the user wears the HMD to prevent over-saturation or under-saturation of the images and the additional images.

Determining fixation of a user's eyes from images of portions of the user's face enclosed by a head mounted display

A virtual reality (VR) or augmented reality (AR) head mounted display (HMD) includes various image capture devices that capture images of portions of the user's face and body. Through image analysis, points of each portion of the user's face and body are identified from the images and their movement is tracked. The identified points are mapped to a three dimensional model of a face and to a three dimensional model of a body. From the identified points, animation parameters describing positioning of various points of the user's face and body are determined for each captured image. From the animation parameters and transforms mapping the captured images to three dimensions, the three dimensional model of the face and the three dimensional model of the body is altered to render movement of the user's face and body.

Generating graphical representation of a user's face and body using a monitoring system included on a head mounted display

An interactive system (e.g., a head-mounted-display device) may include (1) a facial coupling subsystem having electrodes configured to conduct at least one biopotential signal (e.g., an EOG or EMG signal) generated by a user's body, (2) a receiving subsystem electrically connected to the facial coupling subsystem and configured to receive, from the user's body via at least one electrode of the facial coupling subsystem, the biopotential signal, and (3) a detection subsystem electrically connected to the receiving subsystem and configured to (a) determine at least one characteristic of the biopotential signal and (b) use the at least one characteristic of the biopotential signal to determine a gaze direction of an eye of the user and/or a facial gesture of the user. Various other apparatus, systems, and methods are also disclosed.

Apparatus, systems, and methods for sensing biopotential signals

A virtual reality (VR) or augmented reality (AR) head mounted display (HMD) includes various image capture devices that capture images of portions of the user's face. Through image analysis, points of each portion of the user's face are identified from the images and their movement is tracked. The identified points are mapped to a three dimensional model of a face. From the identified points, a blendshape vector is determined for each captured image, resulting in various vectors indicating the user's facial expressions. A direct expression model that directly maps images to blendshape coefficients for a set of facial expressions based on captured information from a set of users may augment the blendshape vector in various embodiments. From the blendshape vectors and transforms mapping the captured images to three dimensions, the three dimensional model of the face is altered to render the user's facial expressions.

Generating graphical representation of facial expressions of a user wearing a head mounted display accounting for previously captured images of the user's facial expressions

A virtual reality (VR) or augmented reality (AR) head mounted display (HMD) includes multiple image capture devices positioned within and on the HMD to capture portions of a face of a user wearing the HMD. Multiple image capture devices are included within the HMD to capture different portions of the face of the user within the HMD, and one or more other image capture devices are positioned to capture portions of the face of the user external to the HMD. Captured images from various image capture devices may be communicated to a console or a controller that generates a graphical representation of the user's face based on the captured images.

Growing community of inventors

Pittsburgh, PA, United States of America

Dawei Wang