Facebook Technologies, LLC

Meta Platforms Technologies, LLC

Oculus, Vr, LLC

Alexander Jobe Fix

Robert Dale Cavin

Douglas Robert Lanman

Andrew John Ouderkirk

Brian M Wheelwright

Hee Yoon Lee

Andrew Maimone

Matthieu Charles Raoul Leibovici

Ganghun Kim

Sebastian Sztuk

Javier San Agustin Lopez

Ying Geng

Scott Robert Ramsby

Mohammadhossein Daraeihajitooei

Mohamed Hegazy

Dmitri Model

Anton S Kaplanyan

Lei Xiao

Matthew Steven Chapman

Andrew Wyman MacDonald

Robin Sharma

Nicholas Daniel Trail

Nathan Seigo Matsuda

Kirk Erik Burgess

Yusufu Njoni Bamaxam Sulai

Scott Charles McEldowney

Karol Constantine Hatzilias

Ryan Michael Ebert

Stephen James McNally

Brett Joseph Bryars

William Aaron Nicholls

Elias Daniel Guestrin

Michael Sean Mounier

Qi Zhang

Sapna A Shroff

Thomas Scott Murdison

Evan Gander

Clare Joyce Robinson

Salah Eddine Nouri

Mohammadhossein Daraeihajitooel

Timo Stoffregen

Alexander Grant Anderson

Techniques are described for generating and using an illumination pattern for corneal topography. In some examples, the illumination pattern is projected over a sequence of frames, including a first frame with a first pattern of dots arranged along a two-dimensional grid, and a second frame with a second pattern of dots arranged along the two-dimensional grid. The first pattern differs from the second pattern with respect to a wavelength of light used to project one or more dots, a position of one or more dots, or both. After the illumination pattern has been determined, it can be projected over the sequence of frames and onto an eye of a user. Reflections of the illumination pattern as projected over the sequence of frames can then be used to calculate an eye model characterizing a topography of a cornea of the eye of the user.

Light patterns for corneal topography

An apparatus, system, and method for detecting glints includes a pair of light sources positioned on a head-mounted frame and driven to illuminate a corneal surface of an eye with differential light signals. An image sensor is positioned on the head-mounted frame to receive reflections of the differential light signals from the corneal surface. An orientation of a cornea (e.g., corneal sphere) of the eye may be determined at least partially based on the reflections of the differential light signals.

Differential illumination for corneal glint detection

A disclosed computer-implemented method may include directing a display device included in a head-mounted display worn by a user to illuminate, via a projection of a line at an illumination time, a portion of a cornea of the user. The method may further include detecting, via an image sensor at a detection time, a portion of the projection of the line reflected by the portion of the cornea of the user and identifying a distortion of the projection of the line reflected by the portion of the cornea of the user. The method may also include determining a shape of the cornea of the user based on the illumination time, the detection time, and the distortion of the projection of the line reflected by the portion of the cornea. Various other methods, systems, apparatuses, and computer-readable media are also disclosed.

Apparatuses, systems, and methods for mapping corneal curvature

The disclosed eye-tracking systems may include a visible light display having visible light pixels, at least one infrared pixel positioned within bounds of the visible light display, and an infrared sensor positioned and configured to detect infrared light originating from the at least one infrared pixel and reflected from an eye of the user. Related head-mounted display systems and methods for eye tracking are also disclosed.

Systems and methods for using a display as an illumination source for eye tracking

Techniques are described for generating and using an illumination pattern for corneal topography. The illumination pattern is projected onto an eye of a user wearing a head-mounted assembly. The illumination pattern is based on a reference pattern and corresponds to selective illumination of dots arranged along a two-dimensional grid. An image sensor captures a reflected image produced by reflection of the illumination pattern off the eye. A reflected pattern is identified based on glints in the reflected image and mapped to the reference pattern to generate an aligned reflected pattern. An eye model including a topography of a cornea is calculated by comparing the aligned reflected pattern to the reference pattern to determine a deviation in a shape of the cornea based on a difference between the aligned reflected pattern and the reference pattern. The eye model can be applied in various ways, including for eye tracking or biometric authentication.

Corneal topography mapping with dense illumination

A system for making a holographic medium for use in generating light patterns for eye tracking includes a light source configured to provide light and a beam splitter configured to separate the light into a first portion of the light and a second portion of the light that is spatially separated from the first portion of the light. The system also includes a first set of optical elements configured to transmit the first portion of the light for providing a first wide-field beam onto an optically recordable medium and one or more diffractive optical elements configured to receive the second portion of the light and project a plurality of separate light patterns onto the optically recordable medium for forming the holographic medium.

Holographic in-field illuminator

Holographic pattern generation for head-mounted display (HMD) eye tracking using a diffractive optical element

In one embodiment, a method includes receiving a first frame associated with a first time and one or more second frames of a video having a resolution lower than a target resolution, wherein each second frame is associated with a second time prior to the first time, generating a first feature map for the first frame and one or more second feature maps for the one or more second frames, up-sampling the first feature map and the one or more second feature maps to the target resolution, warping each of the up-sampled second feature maps according to a motion estimation between the associated second time and the first time, and generating a reconstructed frame having the target resolution corresponding to the first frame by using a machine-learning model to process the up-sampled first feature map and the one or more up-sampled and warped second feature maps.

Neural super-sampling for real-time rendering

A system for making a holographic medium for use in generating light patterns for eye tracking includes a light source configured to provide light and a beam splitter configured to separate the light into a first portion of the light and a second portion of the light that is spatially separated from the first portion of the light. The system also includes a first set of optical elements configured to transmit the first portion of the light for providing a first wide-field beam onto an optically recordable medium and a plurality of optical fibers configured to receive the second portion of the light and project a plurality of separate light patterns onto the optically recordable medium for forming the holographic medium.

Holographic pattern generation for head-mounted display (HMD) eye tracking using a fiber exposure

A system for making a holographic medium for use in generating light patterns for eye tracking includes a light source configured to provide light and a beam splitter configured to separate the light into a first portion of the light and a second portion of the light that is spatially separated from the first portion of the light. The system also includes a first set of optical elements configured to transmit the first portion of the light for providing a first wide-field beam onto an optically recordable medium, a second set of optical elements configured to transmit the second portion of the light for providing a second wide-field beam, and a plurality of parabolic reflectors optically coupled with the second set of optical elements and configured to receive the second wide-field beam and project a plurality of separate light patterns onto the optically recordable medium for forming the holographic medium.

Holographic pattern generation for head-mounted display (HMD) eye tracking using an array of parabolic mirrors

A system predicts future positions or vergence depth of the user's eyes and generates gaze contingent content, such as for a head-mounted display (HMD), based on the predicted positions or vergence depth. The system includes an eye tracking controller that creates eye tracking information defining positions of a first eye and a second eye of a user over time. The eye tracking information is input to a neural network model that outputs the predicted positions or vergence depth. The predicted positions or vergence depth is then used to render the gaze contingent content, or to change other configurations of the HMD. Latency between the detection of user eye movement and the output of corresponding content is reduced to provide a more immersive real-time user experience.

Predictive eyetracking using recurrent neural networks

A disclosed computer-implemented method may include directing an illumination source, positioned at a predetermined position relative to an event camera positioned to receive light from the illumination source reflected by a portion of a cornea of a user during an illumination sequence, to illuminate the portion of the cornea of the user in accordance with the illumination sequence. The method may also include detecting, via the event camera, a set of events, each event in the set of events comprising a change in an intensity of light from the illumination source received by a pixel included in the event camera over a sampling period of the event camera. The method may further include determining a shape of the portion of the cornea of the user based on the predetermined position, the illumination sequence, and the set of events. Various other apparatuses, systems, methods, and computer-readable media are also disclosed.

Apparatuses, systems, and methods for mapping a surface of an eye via an event camera

A disclosed apparatus may include a line source configured to produce a line of light and a scanning device configured to scan the line of light across a scanning field in a scanning direction. The scanning field may include a receiving portion configured to receive an eye. The apparatus may also include a reflector positioned within the scanning field. During a primary period of a scan, the line of light may scan the receiving portion in a primary direction. During a secondary period of the scan, the reflector may reflect the line of light such that a reflection of the line of light scans the receiving portion in a secondary direction. The apparatus may also include a photodetector positioned to receive (1) an initial reflection during the primary period, and (2) a subsequent reflection during the secondary period. Various other methods, apparatuses, and computer-readable media are also disclosed.

Apparatuses, systems, and methods for scanning an eye via a folding mirror

In one embodiment, a computing system may receive current eye-tracking data associated with a user of a head-mounted display. The system may dynamically adjust a focal length of the head-mounted display based on the current eye-tracking data. The system may generate an in-focus image of a scene and a corresponding depth map of the scene. The system may generate a circle-of-confusion map for the scene based on the depth map. The circle-of-confusion map encodes a desired focal surface in the scene. The system may generate, using a machine-learning model, an output image with a synthesized defocus-blur effect by processing the in-focus image, the corresponding depth map, and the circle-of-confusion map of the scene. The system may display the output image with the synthesized defocus-blur effect to the user via the head-mounted display having the adjusted focal length.

Systems and methods for generating defocus blur effects

Growing community of inventors

Seattle, WA, United States of America

Alexander Jobe Fix