Google Inc.

North Corporation

Thalmic Labs Inc.

Facebook Technologies, LLC

Idris S Aleem

Mayank Bhargava

Andrew S Logan

Rees Anwyl Samuel Simmons

Pedram Ataee

Matthew Bailey

Jake Chapeskie

Mélodie Vidal

Sushant Kulkarni

Ahmed Gawish

Daniel David Albin Balzerson

Stephen Lake

Philip L Davidson

Kevin R Moule

Sui Tong Tang

Bergen Fletcher

Cecilia Abadie

Zhiheng Jia

Lukas Rezek

Samuel Jewell Lochner

Dylan Jacobs

Rees Simmons

Cory Stegelmeier

Ehsan Parvizi

A system and method of detecting display fit measurements and/or ophthalmic measurements for a head mounted wearable computing device including a display device is provided. An image of a fitting frame worn by a user of the computing device is captured by the user, through an application running on the computing device. One or more keypoints and/or features and/or landmarks are detected in the image including the fitting frame. A three-dimensional pose of the fitting frame is determined based on the detected keypoints and/or features and/or landmarks, and configuration information associated with the fitting frame. The display device of the head mounted wearable computing device can then be configured based on the three-dimensional pose of the fitting frame as captured in the image.

Image based detection of fit for a head mounted wearable computing device

A system and method of detecting display fit measurements and/or ophthalmic measurements for a head mounted wearable computing device including a display device is provided. An image of a fitting frame worn by a user of the computing device is captured by the user, through an application running on the computing device. One or more visual markers each including a distinct pattern are detected in the image including the fitting frame. A model of the fitting frame, and configuration information associated with the fitting frame, are determined based on the detection of the pattern. A three-dimensional pose of the fitting frame is determined based on the detected visual marker(s) and patterns, and the configuration information associated with the fitting frame. The display device of the head mounted wearable computing device can then be configured based on the three-dimensional pose of the fitting frame as captured in the image.

Image based detection of display fit and ophthalmic fit measurements

Systems and methods for validation of modeling and simulation systems that provide for the virtual try-on of wearable devices, such as glasses, by a user, and for the virtual fitting of selected wearable devices for the user. Wearable fit measurements, display fit measurements, ophthalmic fit measurements and other such measurements associated with the fit and function of the wearable device may be detected from image data capturing the wearable device worn by the user. The detected measurements may be compared to corresponding measurements detected in a virtual simulation of the wearable device worn by the user. The comparison may provide for validation and increased accuracy/realism of the modeling and simulation systems.

Validation of modeling and simulation of virtual try-on of wearable device

A system and method of detecting display fit measurements and/or ophthalmic measurements for a head mounted wearable computing device including a display device is provided. The system and method may include capturing image data including a face of a user to be fitted for the head mounted wearable computing device. A three-dimensional head pose and gaze measurements may be extracted and a three-dimensional model may be developed from the captured image data. The system may detect display fit measurements and/or ophthalmic fit measurements from the three-dimensional model, and may provide one or more head mounted wearable computing devices that meet the display fit and/or ophthalmic fit requirements.

Predicting display fit and ophthalmic fit measurements using a simulator

A method performed by a server can include sending a first webpage to a first computing device, the first computing device including a camera, the first webpage including an image-capturing function and including an instruction for a user to obtain a second webpage via a second computing device, the second webpage including a calibration image file; receiving, from the first computing device, multiple captured images that were captured by the camera, the multiple calibration images including instances of a calibration image presented by a display included in the second computing device, the calibration image being a representation of the calibration image file; and based on the multiple captured images, calibrating the camera.

Calibrating camera in electronic device

Systems and methods for validation of modeling and simulation systems that provide for the virtual fitting of wearable devices, such as glasses, by a user. Three-dimensional modeling and simulation of test subject both with and without fitting frames corresponding to a wearable device may be captured to validate the modeling and simulation modules and associated algorithms and machine learning modules used to simulate the fit of the wearable device on a user. Validation in this manner may provide for increased accuracy/realism of the modeling and simulation systems.

Validation of modeling and simulation of wearable device

A method includes receiving a face image and lens measurements of a person, and selecting a three dimensional (3-D) model of a frame of a pair of smart glasses from a plurality of 3-D models of frames for placement on the face image of the person. The pair of smart glasses includes a virtual display embedded in, or overlaid on a lens. The method further includes sizing the selected 3-D model of the frame of the pair of smart glasses for placement on the face image of person based on the lens measurements of the person, and determining a gazability of the virtual display in the 3-D model of the pair of smart glasses placed on the face image of the person.

Determining display gazability and placement of virtual try-on glasses using optometric measurements

A system and method are provided for sizing and fitting a head mounted wearable computing device for a user based on image data of the head of the user, including a known reference device having a known scale. The system and method may include capturing image data including a face of the user to be fitted for the head mounted wearable computing device. The known reference device having the known scale is compared to features detected in the image data to determine a scaling factor. The scaling factor is used to size, or assign measures to facial features detected in the image data. A three-dimensional model of the head of the user may be generated from the captured image data.

Image-based fitting of a wearable computing device

According to an aspect, a method for designing an eye tracker on a wearable device includes selecting a first three-dimensional (3D) model of at least a head of a person, selecting a second 3D model of a wearable device, positioning a synthetic eye within the first 3D model, positioning an eye tracker component at a first location on the second 3D model, the eye tracker component including at least one of an eye tracker sensor or a light source, moving at least one of the first 3D model or the second 3D model such that at least a portion of the first 3D model contacts at least a portion of the second 3D model, and generating performance data with the eye tracker component positioned at the first location on the second 3D model.

Eye tracker design for a wearable device

A method of tracking an eye of a user includes generating an infrared light, scanning the infrared light over the eye, and detecting reflections of the infrared light from the eye over an eye tracking period. A plurality of glints is identified from the reflections of the infrared light detected. A glint center position of each glint in a glint space is determined and transformed to a gaze position in a display space. At least once during the eye tracking period, an image of the eye is reconstructed from a portion of the reflections of the infrared light detected. A pupil is detected from the image, and a pupil center position is determined. A glint-pupil vector is determined from the pupil center position and the glint center position of at least one glint corresponding in space to the pupil. The glint space is recalibrated based on the glint-pupil vector.

Method and system for eye tracking with glint space recalibration on wearable heads-up display

According to an aspect, a method includes receiving a first three-dimensional model (3D) model of at least a body part of a person, receiving a second 3D model of a wearable device, and predicting, by at least one machine-learning (ML) model, a plurality of contact points between the first 3D model and the second 3D model.

Prediction of contact points between 3D models

A wearable heads-up display (WHUD) obtains attribute data corresponding to an attribute of an item of interest and obtains environmental data of an environment surrounding the WHUD via one or more sensors of the WHUD. The WHUD compares the attribute data with the environmental data to detect the item of interest. In response to the detection, the WHUD obtains location data indicative of a location of the item of interest, stores the location data in association with a context of detection of the item of interest. In response to a trigger, such as a query by a user regarding the item of interest, the WHUD provides a location indication based on the location data, the location indication including, for example, a display of a description of the location of the item of interest, a display of the item of interest at the location, and the like.

Tracking of item of interest using wearable heads up display

An ear saddle point of a subject's ear is determined to produce specifications to fit a wearable apparatus to a subject's head. A boundary of the subject's ear in a profile image is determined using a first model. A map of the subject's ear is generated indicating probable locations of a two-dimensional ear saddle point. A most probable location of the two-dimensional ear saddle point is determined based on the probability map. The two-dimensional ear saddle point is projected onto a three-dimensional mesh surface representing the subject's head. A maximum depth of the three-dimensional mesh surface is determined in a defined region around the projected two-dimensional ear saddle point. The ear saddle point is computed based on the projected two-dimensional ear saddle point and the determined maximum depth. Specifications of a wearable apparatus are generated to fit the apparatus to the subject's head based on the computed ear saddle point.

Systems and methods for determining an ear saddle point of a user to produce specifications to fit a wearable apparatus to the user's head

A method of tracking a gaze of an eye includes tracking the gaze of the eye in a first tracking mode. Glint data of at least one glint of the eye is obtained during at least a portion of tracking the gaze of the eye in the first tracking mode. The glint data is in a time domain and includes a time series of a spatial descriptor associated with the at least one glint. The glint data is transformed into a frequency domain to generate a glint frequency spectrum. A stability of the at least one glint is determined based on the glint frequency spectrum. If the at least one glint is determined to be unstable, tracking of the gaze of the eye is switched from the first tracking mode to a second tracking mode that is different from the first tracking mode.

Growing community of inventors

Kitchener, Canada

Idris S Aleem