Massachusetts Institute of Technology

Leland Stanford Junior University

Sesame Ai, Inc.

Google Inc.

Magic Leap, Inc.

Zinn Labs, Inc.

University of California

Dolby Laboratories Licensing Corporation

King Abdullah University of Science and Technology

Raxium, Inc.

Gordon Wetzstein

Ramesh Raskar

Robert Konrad

Nitish Padmanaban

Wolfgang Heidrich

Andrew Victor Jones

Kari Pulli

Ryan Phillip Spicer

Brian T Schowengerdt

Michael Anthony Klug

Matthew Waggener Hirsch

Michal Beau Dennison Vaughn

Kevin Boyle

Tomi Petteri Maila

Douglas Robert Lanman

Felix Heide

John Gabriel Buckmaster

Karl A Deisseroth

Mark Luitzen Brongersma

Dianmin Lin

James Gregson

Pieter G Kik

Isaac V Kauvar

Anastasios Nikolas Angelopoulos

Vincent Lee

Ivo Bodo Ihrke

Matthias Hullin

Julien Martel

Fu-Chung Huang

Tim Machado

Petr Kellnhofer

Lars Jebe

Zisimos Economou

Kshitij Marwah

Brian Barsky

Michal Beau Vaughn

A tracking system for object detection and tracking. The system may include a plurality of light sources, a differential camera, and a controller. The plurality of light sources is positioned at different locations on a device and are configured to emit pulses of light that illuminate an object. The differential camera has an optical axis, and at least some of the plurality of light sources are off-axis relative to the optical axis. The differential camera is configured to detect a change in brightness of the object caused in part by one or more of the pulses of light, and asynchronously output data samples corresponding to the detected change in brightness. The controller is configured to track the object based in part on the data samples output by the differential camera.

Object tracking system using a differential camera

Embodiments are related to a headset integrated into a healthcare platform. The headset comprises one or more sensors embedded into a frame of the headset, a controller coupled to the one or more sensors, and a transceiver coupled to the controller. The one or more sensors capture health information data for a user wearing the headset. The controller pre-processes at least a portion of the captured health information data to generate a pre-processed portion of the health information data. The transceiver communicates the health information data and the pre-processed portion of health information data to an intermediate device communicatively coupled to the headset. The intermediate device processes at least one of the health information data and the pre-processed portion of health information data to generate processed health information data for a health-related diagnostic of the user.

Headset integrated into healthcare platform

A system enables a user to query based on a user's gaze by receiving a query from the user and capturing, via an eye tracking system on a headset, the user's gaze location near an object in a local area. The system captures one or more images of the local area with the object and formats the images based in part on a region of interest in the one or more images that includes the object. The system generates a formatted query based in part on the query. The formatted query is provided to a search engine. Information describing the object determined from the one or more formatted images and information describing the query are used by the search engine to determine an answer to the query about the object. The system presents the answer to the query about the object.

Gaze assisted search query

A differential camera system for object tracking. The system includes a co-aligned light source camera assembly (LSCA) and a controller. The co-aligned LSCA includes a light source and a differential camera sensor. The light source is configured to emit light along an optical path that is directed towards an eye box including an eye of a user. The differential camera sensor is configured to detect a change in brightness of the eye caused in part by the emitted light, asynchronously output data samples corresponding to the detected change in brightness, wherein the optical path is substantially co-aligned with an optical path of the differential camera sensor. The controller is configured to identify a pupil of the eye based on data samples output from the differential camera sensor resulting from the emitted light, and determine a gaze location of the user based in part on the identified pupil.

Tracking system using a differential camera with a co-aligned light source assembly

Systems and methods for event-based gaze in accordance with embodiments of the invention are illustrated. One embodiment includes an event-based gaze tracking system, including a camera positioned to observe an eye, where the camera is configured to asynchronously sample a plurality of pixels to obtain event data indicating changes in local contrast at each pixel in the plurality of pixels, and a processor communicatively coupled to the camera, and a memory communicatively coupled to processor, where the memory contains a gaze tracking application, where the gaze tracking application directs the processor to, receive the event data from the camera, fit an eye model to the eye using the event data, map eye model parameters from the eye model to a gaze vector, and provide the gaze vector.

Systems and methods for event-based gaze tracking

The disclosure describes aspects of a display processing circuitry. In an aspect, one or more displays that support multiple views include one or more arrays of pixels, one or more backplanes, and a processing circuitry configured to receive one or more data streams, control processing of the data streams based on policies from which to select a mode of operation, each mode of operation defining which rays of light the arrays of pixels in the displays are to contribute to generate a particular view or views and the tasks to be performed by the processing circuitry to modify the data streams accordingly. The processing circuitry further provides signaling representative of the modified data streams to the arrays of pixels through a circuit configuration of the backplanes for the arrays of pixels to contribute the rays that will to generate the particular view or views. A corresponding method is also described.

Display processing circuitry

Disclosed herein is methods and systems for providing different views to a viewer. One particular embodiment includes a method including providing, to a neural network, a plurality of 2D images of a 3D object. The neural network may include a signed distance function based sinusoidal representation network. The method may further include obtaining a neural model of a shape of the object by obtaining a zero-level set of the signed distance function; and modeling an appearance of the object using a spatially varying emission function. In some embodiments, the neural model may be converted into a triangular mesh representing the object which may be used to render multiple view-dependent images representative of the 3D object.

Methods and systems for rendering view-dependent images using 2D images

Provided herein is a macroscope comprising an objective apparatus comprising a multifocal widefield optics comprising a plurality of optical components configured to focus on a plurality of planes. Also provided herein are methods for analyzing a three-dimensional specimen, the method comprising obtaining, via a macroscope, synchronous multifocal optical images of a plurality of planes of the three-dimensional specimen, wherein the macroscope comprises an objective apparatus comprising a multifocal widefield optics comprising a plurality of optical components configured to focus on a plurality of planes. The three-dimensional specimen can be a biological specimen, such as brain.

Multifocal macroscope for large field of view imaging of dynamic specimens

An augmented reality display system is configured to direct a plurality of parallactically-disparate intra-pupil images into a viewer's eye. The parallactically-disparate intra-pupil images provide different parallax views of a virtual object, and impinge on the pupil from different angles. In the aggregate, the wavefronts of light forming the images approximate a continuous divergent wavefront and provide selectable accommodation cues for the user, depending on the amount of parallax disparity between the intra-pupil images. The amount of parallax disparity is selected using a light source that outputs light for different images from different locations, with spatial differences in the locations of the light output providing differences in the paths that the light takes to the eye, which in turn provide different amounts of parallax disparity. Advantageously, the wavefront divergence, and the accommodation cue provided to the eye of the user, may be varied by appropriate selection of parallax disparity, which may be set by selecting the amount of spatial separation between the locations of light output.

System and method for presenting image content on multiple depth planes by providing multiple intra-pupil parallax views

A device includes a camera assembly and a controller. The camera assembly is configured to capture images of both eyes of a user. Using the captured images, the controller determines a location for each pupil of each eye of the user. The determined pupil locations and captured images are used to determine eye tracking parameters which are used to compute values of eye tracking functions. With the computed values and a model that maps the eye tracking functions to gaze depths, a gaze depth of the user is determined. An action is performed based on the determined gaze depth.

Determining gaze depth using eye tracking functions

Embodiments are related to a plurality of gaze sensors embedded into a frame of a headset for detection of a gaze vector of a user wearing the headset and user's control at the headset. The gaze vector for an eye of the user can be within a threshold distance from one of the gaze sensors. By monitoring signals detected by the gaze sensors, it can be determined that the gaze vector is within the threshold distance from the gaze sensor. Based on this determination, at least one action associated with the headset is initiated.

Gaze sensors and display elements for detection of gaze vectors and user control at headset

A multiview autostereoscopic display includes a display area including an array of angular pixels, an eye tracker, and a processing system. Each angular pixel emits color that varies across a field of view of that angular pixel. The array of angular pixels displays different views in different viewing zones across the field of view of the display. The eye tracker detects the presence of the eyes of at least one viewer within specific viewing zones and produces eye tracking information including locations of the detected eyes within the specific viewing zones. The processing system renders a specific view for each detected eye based upon the location of the detected eye within the viewing zone with detected eyes, and generates control information for the array of angular pixels to cause the specific view for each detected eye to be displayed in the viewing zone in which that eye was detected.

Light field displays incorporating eye trackers and methods for generating views for a light field display using eye tracking information

Systems and methods for imaging object velocity are provided. In an embodiment, at least one Time-of-Flight camera is used to capture a signal representative of an object in motion over an exposure time. Illumination and modulation frequency of the captured motion are coded within the exposure time. A change of illumination frequency is mapped to measured pixel intensities of the captured motion within the exposure time, and information about a Doppler shift in the illumination frequency is extracted to obtain a measurement of instantaneous per pixel velocity of the object in motion. The radial velocity information of the object in motion can be simultaneously captured for each pixel captured within the exposure time. In one or more aspects, the illumination frequency can be coded orthogonal to the modulation frequency of the captured motion. The change of illumination frequency can correspond to radial object velocity.

Doppler time-of-flight imaging

Growing community of inventors

Palo Alto, CA, United States of America

Gordon Wetzstein