Magic Leap, Inc.

Northrop Grumman Systems Corporation

Applied Minds, Inc.

Applied Invention, LLC

Intel Corporation

Adrian Kaehler

Gary R Bradski

Vijay Badrinarayanan

Nicole Elizabeth Samec

Gholamreza Amayeh

Mark Baerenrodt

Christopher M Harrises

Bran Ferren

Kjerstin Irja Williams

Nastasja U Robaina

Michael Keesling

David Beal

Pablo Maurin

Dan Ruderman

Eric Powers

Michael Anthony Klug

Eric Baerenrodt

Tammy Sherri Powers

Alexey Spizhevoy

Shi-Ping Hsu

Adam Carl Wright

Douglas Bertram Lee

Mark E Duttweiler

Ralph Remsburg

Evan Francis Rynk

Guillermo Padin Rohena

W Daniel Hillis

Ivan Li Chuen Yeoh

H Keith Nishihara

Russel Howe

Jeffrey Scott Sommers

Thomas Anthony Tombrello, Jr

Nitin Singh

James Wallace Sarrett

Jennifer M R Devine

Luke Walter Khanlian

John Adam Croston

Joseph Wayne Seuck

Douglas K Lee

Lars Jangaard

Doug Lee

Prasanna Krishnasamy

Natasja U Robaina

Fred Zyda

Matt Stine

Dan Selnick

Eric Gradman

Sam Leventer

Andy Spix

Sunil Sankar

Mike Reitsma

Disclosed herein is a wearable display system for capturing retraining eye images of an eye of a user for retraining a neural network for eye tracking. The system captures retraining eye images using an image capture device when user interface (UI) events occur with respect to UI devices displayed at display locations of a display. The system can generate a retraining set comprising the retraining eye images and eye poses of the eye of the user in the retraining eye images (e.g., related to the display locations of the UI devices) and obtain a retrained neural network that is retrained using the retraining set.

Personalized neural network for eye tracking

An augmented reality device (ARD) can present virtual content which can provide enhanced experiences with the user's physical environment. For example, the ARD can detect a linkage between a person in the FOV of the ARD and a physical object (e.g., a document presented by the person) or detect linkages between the documents. The linkages may be used in identity verification or document verification.

Augmented reality identity verification

A wearable device can include an inward-facing imaging system configured to acquire images of a user's periocular region. The wearable device can determine a relative position between the wearable device and the user's face based on the images acquired by the inward-facing imaging system. The relative position may be used to determine whether the user is wearing the wearable device, whether the wearable device fits the user, and/or whether an adjustment to a rendering location of a virtual object can be made to compensate for a deviation of the wearable device from its normal resting position relative to the user's face.

Periocular test for mixed reality calibration

Systems and methods for generating a face model for a user of a head-mounted device are disclosed. The head-mounted device can include one or more eye cameras configured to image the face of the user while the user is putting the device on or taking the device off. The images obtained by the eye cameras may be analyzed using a stereoscopic vision technique, a monocular vision technique, or a combination, to generate a face model for the user. The face model can be used to generate a virtual image of at least a portion of the user's face, for example to be presented as an avatar.

Face model capture by a wearable device

Systems and methods for eye image segmentation and image quality estimation are disclosed. In one aspect, after receiving an eye image, a device such as an augmented reality device can process the eye image using a convolutional neural network with a merged architecture to generate both a segmented eye image and a quality estimation of the eye image. The segmented eye image can include a background region, a sclera region, an iris region, or a pupil region. In another aspect, a convolutional neural network with a merged architecture can be trained for eye image segmentation and image quality estimation. In yet another aspect, the device can use the segmented eye image to determine eye contours such as a pupil contour and an iris contour. The device can use the eye contours to create a polar image of the iris region for computing an iris code or biometric authentication.

Neural network for eye image segmentation and image quality estimation

An augmented reality (AR) device can be configured to monitor ambient audio data. The AR device can detect speech in the ambient audio data, convert the detected speech into text, or detect keywords such as rare words in the speech. When a rare word is detected, the AR device can retrieve auxiliary information (e.g., a definition) related to the rare word from a public or private source. The AR device can display the auxiliary information for a user to help the user better understand the speech. The AR device may perform translation of foreign speech, may display text (or the translation) of a speaker's speech to the user, or display statistical or other information associated with the speech.

Word flow annotation

A wearable device can include an inward-facing imaging system configured to acquire images of a user's periocular region. The wearable device can determine a relative position between the wearable device and the user's face based on the images acquired by the inward-facing imaging system. The relative position may be used to determine whether the user is wearing the wearable device, whether the wearable device fits the user, or whether an adjustment to a rendering location of a virtual object should be made to compensate for a deviation of the wearable device from its normal resting position.

Disclosed herein is an augmented reality (AR) system that provides information about purchasing alternatives to a user who is about to purchase an item or product (e.g., a target product) in a physical retail location. In some variations, offers to purchase the product and/or an alternative product are provided by the merchant and/or competitors via the AR system. An offer negotiation server (ONS) aggregates offer data provided various external parties (EPs) and displays these offers to the user as the user is considering the purchase of a target product. In some variations, an AR system may be configured to facilitate the process of purchasing items at a retail location.

Augmented reality systems and methods for purchasing

A display system comprises a wearable display device for displaying augmented reality content. The display device comprises a display area comprising light redirecting features that are configured to direct light to a user. The display area is at least partially transparent and is configured to provide a view of an ambient environment through the display area. The display device is configured to determine that a reflection of the user is within the user's field of view through the display area. After making this determination, augmented reality content is displayed in the display area with the augmented reality content augmenting the user's view of the reflection. In some embodiments, the augmented reality content may overlie on the user's view of the reflection, thereby allowing all or portions of the reflection to appear to be modified to provide a realistic view of the user with various modifications made to their appearance.

Augmented reality systems and methods utilizing reflections

Apparatus and methods for displaying an image by a rotating structure are provided. The rotating structure can comprise blades of a fan. The fan can be a cooling fan for an electronics device such as an augmented reality display. In some embodiments, the rotating structure comprises light sources that emit light to generate the image. The light sources can comprise light-field emitters. In other embodiments, the rotating structure is illuminated by an external (e.g., non-rotating) light source.

Fan assembly for displaying an image

Examples of an imaging system for use with a head mounted display (HMD) are disclosed. The imaging system can include a forward-facing imaging camera and a surface of a display of the HMD can include an off-axis diffractive optical element (DOE) or hot mirror configured to reflect light to the imaging camera. The DOE or hot mirror can be segmented. The imaging system can be used for eye tracking, biometric identification, multiscopic reconstruction of the three-dimensional shape of the eye, etc.

Eye imaging with an off-axis imager

Systems and methods for generating a face model for a user of a head-mounted device are disclosed. The head-mounted device can include one or more eye cameras configured to image the face of the user while the user is putting the device on or taking the device off. The images obtained by the eye cameras may be analyzed using a stereoscopic vision technique, a monocular vision technique, or a combination, to generate a face model for the user.

Apparatuses and methods for displaying a 3-D representation of an object are described. Apparatuses can include a rotatable structure, motor, and multiple light field sub-displays disposed on the rotatable structure. The apparatuses can store a light field image to be displayed, the light field image providing multiple different views of the object at different viewing directions. A processor can drive the motor to rotate the rotatable structure and map the light field image to each of the light field sub-displays based in part on the rotation angle, and illuminate the light field sub-displays based in part on the mapped light field image. The apparatuses can include a display panel configured to be viewed from a fiducial viewing direction, where the display panel is curved out of a plane that is perpendicular to the fiducial viewing direction, and a plurality of light field sub-displays disposed on the display panel.

Display for three-dimensional image

Wearable spectroscopy systems and methods for identifying one or more characteristics of a target object are described. Spectroscopy systems may include a light source configured to emit light in an irradiated field of view and an electromagnetic radiation detector configured to receive reflected light from a target object irradiated by the light source. One or more processors of the systems may identify a characteristic of the target object based on a determined level of light absorption by the target object. Some systems and methods may include one or more corrections for scattered and/or ambient light such as applying an ambient light correction, passing the reflected light through an anti-scatter grid, or using a time-dependent variation in the emitted light.

Augmented reality system and method for spectroscopic analysis

A wearable display system includes a mixed reality display for presenting a virtual image to a user, an outward-facing imaging system configured to image an environment of the user, and a hardware processor operably coupled to the mixed reality display and to the imaging system. The hardware processor is programmed to generate a virtual remote associated with a parent device, render the virtual remote and the virtual control element on the mixed reality display, determine when the user of the wearable system interacts with the virtual control element of the virtual remote, and perform certain functions in response to user interaction with a virtual control element of the virtual remote. These functions may include generation the virtual control element to move on the mixed reality display; and when movement of the virtual control element surpasses a threshold condition, generate a focus indicator for the virtual control element.

Virtual user input controls in a mixed reality environment

Growing community of inventors

Los Angeles, CA, United States of America

Adrian Kaehler