Facebook Technologies, LLC

Microsoft Technology Licensing, LLC

Meta Platforms Technologies, LLC

Oculus, Vr, LLC

Other

Amazon Technologies, Inc.

Fenwick & West Llp

Michael Anthony Hall

Qing Chao

Xinqiao Liu

Zhaoming Zhu

Andrew Lovitt

Fengqiang Li

Zihe Gao

Shuochen Su

Augusto Ronchini Ximenes

Huixuan Tang

Rajesh Lachhmandas Chhabria

Byron Taylor

Jason Ryan Farmer

Zhengyou Zhang

Andrew David Wilson

Nicholas Daniel Trail

Nikolai Smolyanskiy

Jeremiah Spradlin

Ruonan Yang

Monica Stewart

Desney S Tan

Matthew Graham Dyor

Robert Dale Cavin

Michael Koenig

Tin Komljenovic

Algird Michael Gudaitis

Travis Perry

Peter N Glaskowsky

Ira Snyder

Vikram Reddy Dendi

Steven P Maillet

Mats Eric Myrberg

Spencer Dean Perreault

Kimberly Denise Auyang Hallman

Alexander T Bussmann

Robin Sharma

Cyrus Soli Bamji

Tommer Leyvand

Daniel Reed

Joshua Owen Miller

Nadav Grossinger

James R Larus

Robert Nasry Hasbun

Richard Andrew Newcombe

Tanmoy Dutta

Jered D Aasheim

Daniel Gilbert Kennett

Jonathan Ross Hoof

Steven John Lovegrove

Dominique Fortier

Guohua Wei

Robert Mitchell Smith

Kameron Wade Rausch

Jonatan Ginzburg

Karen Y Cheng

Dennis Tighe

Sean Eron Anderson

Mirko Schmidt

Julian Straub

Abdelrehim Ahmed

Emad Barsoum

David Charles Wein

Gordon George Free

Beibei Cheng

Akshay Johar

David Vakrat

Grant Alexander McAlister

Matthew Adam Simari

Elias Daniel Guestrin

Stefanie Lyn Tomko

Mike Fenton

Greg Shirakyan

Ben Walker

Dave Rohn

Seth Michael Hirsh

Timothy Michael Galvin

Shawn McCoy

Khaled Sinno

Shengqiong Xie

Charles Patrick Saunders

Nathan M Rau

A depth camera assembly (DCA) optimizes illumination and image capture of a local area to generate depth information of the local area. The DCA determines depth information for a first portion of the local area viewable at a first pose. The DCA is moved from the first pose to a second pose, where a second portion of the local area is viewable and overlaps with the first portion. The overlapping region is not illuminated by the DCA. A non-overlapping portion of the second portion is illuminated, captured, and depth information determined.

Dynamic illumination control for depth determination

A depth camera assembly (DCA) for multimode time-of-flight based depth sensing is presented herein. The DCA includes a projector, a detector, and a controller. The projector illuminates a target area with outgoing light comprising a plurality of light pulses. The detector includes an array of unit cells. Each unit cell includes a macropixel with a plurality of pixels that captures portions of the outgoing light reflected from the target area, and an array of memory cells coupled to the macropixel. At least one of the memory cells stores information about the captured portions of the reflected outgoing light received from the macropixel. The controller determines depth information for the target area based in part on data read from the at least one memory cell.

Multimode detector for different time-of-flight based depth sensing modalities

A depth camera assembly (DCA) includes a direct time of flight system for determining depth information for a local area. The DCA includes an illumination source, a camera, and a controller. The illumination source projects light (e.g., pulse of light) into the local area. The camera detects reflections of the projected light from objects in the local area. Using an internal gating selection procedure, the controller selects a gate window that is likely to be associated with reflection of a pulse of light from an object. The selected gate may be used for depth determination. The internal gating selection procedures may be achieved through external target location and selection or through internal self-selection.

Autonomous gating selection to reduce noise in direct time-of-flight depth sensing

Each pixel of a plurality of pixels captures a portion of outgoing light illuminating a local area reflected from the one or more objects in the local area. A flag is generated via a flag determination logic circuit coupled to each pixel of the plurality of pixels. The flag indicates whether the reflected light was captured within a threshold amount of time from a current time. A time to digital converter is enabled via a flag determination logic circuit. The time to digital converter is associated with each pixel of the plurality of pixels to generate a digital representation of time when each pixel of the plurality of pixels captured the reflected light. Depth information is determined for the one or more objects in the local area based in part on the generated flag and the digital representation of time.

Detector with synchronous event driven readout of pixels for direct time-of-flight depth sensing

A depth camera assembly (DCA) determines depth information for a local area. The DCA includes a camera assembly and at least one illuminator. The DCA may select a subset of the VCSELs to provide illumination at any given time. The illuminator may comprise near-field VCSELs configured to generate a structured light (SL) pattern for depth sensing in the near-field and far-field VCSELs configured to generate a SL pattern for depth sensing in the far-field. The near-field VCSELs may comprise a linear emission region which is shorter than a linear emission region of the far-field VCSELs. The DCA may generate and phase shift a quasi-sinusoidal SL pattern. The DCA may phase shift the quasi-sinusoidal SL pattern by alternating which traces on the illuminator are active.

VCSEL arrays for generation of linear structured light features

A depth camera assembly (DCA) includes a light generator emitting a beam of light into a local area and a detector. The detector captures light from the beam reflected by objects in the local area to a portion of an array of pixels that each include a single photon avalanche diode (SPAD). The location of the portion of the array is based in part on the angle of the beam emitted from the projector. The DCA identifies a set of pixels of the array corresponding to the portion and selectively retrieves current generated from the reflected light by the pixels in the portion of the array without retrieving current generated by pixels in other portions of the array.

Depth sensing using a time of flight system including a scanning beam in combination with a single photon avalanche diode array

A depth camera assembly (DCA) for depth sensing of a local area. The DCA includes a transmitter, a receiver, and a controller. The transmitter illuminates a local area with outgoing light in accordance with emission instructions. The transmitter includes a fine steering element and a coarse steering element. The fine steering element deflects one or more optical beams at a first deflection angle to generate one or more first order deflected scanning beams. The coarse steering element deflects the one or more first order deflected scanning beams at a second deflection angle to generate the outgoing light projected into the local area. The receiver captures one or more images of the local area including portions of the outgoing light reflected from the local area. The controller determines depth information for one or more objects in the local area based in part on the captured one or more images.

Non-mechanical beam steering assembly

Embodiments relate to tracking and determining a location of an object in an environment surrounding a user. A system includes one or more imaging devices and an object tracking unit. The system identifies an object in a search region, determines a tracking region that is smaller than the search region corresponding to the object, and scans the tracking region to determine a location associated with the object. The system may generate a ranking of objects, determine locations associated with the objects, and generate a model of the search region based on the locations associated with the objects.

Object tracking assisted with hand or eye tracking

A depth measurement assembly (DMA) includes a pulsed illuminator assembly, a depth camera assembly, and a controller. The pulsed illuminator assembly has a structured light projector that projects pulses of structured light at a pulse rate into a local area. The depth camera assembly captures images data of an object in the local area illuminated with the pulses of structured light. An exposure interval of the depth camera assembly is pulsed and synchronized to the pulses projected by the pulsed illuminator assembly. The controller controls the pulsed illuminator assembly and the depth camera assembly so that they are synchronized. The controller also determine depth and/or tracking information of the object based on the captured image data. In some embodiments, the pulsed illuminator assembly have a plurality of structured light projectors that projects pulses of structured light at different times.

Depth measurement using a pulsed structured light projector

A depth camera assembly is used to obtain depth information describing a local area. The depth camera assembly includes a sensor having a plurality of pixels. Some or all of the pixels are divided into groups that are coupled to respective multi-purpose time-to-digital converters. Each multi-purpose time-to-digital converter comprises an oscillator and a counter. Each pixel in a group is associated with a multiplexer that is configured to select between inputs coupled to the pixel, the oscillator, or a counter associated with a different pixel. The multiplexer outputs the signal to the counter for the time-to-digital converter associated with the pixel. A time-of-flight measurement is taken during a first portion of an image frame. During a second portion of the image frame, the sensor may be used as an intensity counter. During a third portion of the image frame, the depth sensor may be calibrated.

Time-to-digital converter for depth sensing

A depth camera assembly (DCA) determines depth information. The DCA projects a dynamic structured light pattern into a local area and captures images including a portion of the dynamic structured light pattern. The DCA determines regions of interest in which it may be beneficial to increase or decrease an amount of texture added to the region of interest using the dynamic structured light pattern. For example, the DCA may identify the regions of interest based on contrast values calculated using a contrast algorithm, or based on the parameters received from a mapping server including a virtual model of the local area. The DCA may selectively increase or decrease an amount of texture added by the dynamic structured light pattern in portions of the local area. By selectively controlling portions of the dynamic structured light pattern, the DCA may decrease power consumption and/or increase the accuracy of depth sensing measurements.

Dynamic adjustment of structured light for depth sensing systems based on contrast in a local area

A depth camera assembly (DCA) determines distances between the DCA and objects in a local area within a field of view of the DCA. The DCA projects a series of sinusoidal patterns into the local area DCA and captures images of the sinusoidal patterns via a sensor. Each pixel of the augmented sensor includes a plurality of charge bins, and charge accumulated by a photodiode of a pixel during different time intervals (e.g., times when different sinusoidal patterns are emitted) is stored in a different charge storage bin. Charge may be retrieved from different charge storage bins to determine depth from the DCA.

Timing of multiplexed sensor phase measurements in a depth camera assembly for depth determination using fringe interferometry

A depth estimation system is described capable of determining depth information using two images from two cameras. A first camera captures a first image and a second camera captures a second image, both images including a plurality of light channels. A scan direction is selected from a plurality of scan directions. For the selected scan direction, along each of a plurality of scanlines, the system compares pixels from the first image to pixels from the second image. The comparison is based on calculating a census transform for each pixel in the first image and a census transform for each pixel in the second image. This comparison is used to determine a stereo correspondence between the pixels in the first image and the pixels in the second image. The system generates a depth map based on the stereo correspondence.

Multi-channel depth estimation using census transforms

A depth camera assembly (DCA) includes a direct time of flight system for determining depth information for a local area. The DCA includes an illumination source, a camera, and a controller. In some embodiments, the controller uses previous image frames to determine confidence measurements, and selectively adjusts a number of pulses from the illuminator in a subsequent frame based on the determined confidence values. In some embodiments, the sensor uses autonomous gating, and the depth system includes a depth recovery pipeline which provide depth map estimates from the autonomous gated measurements.

A depth measurement assembly (DMA) includes an illumination source that projects pulses of light (e.g., structured light) at a temporal pulsing frequency into a local area. The DMA includes a sensor that capture images of the pulses of light reflected from the local area and determines, using one or more of the captured images, one or more TOF phase shifts for the pulses of light. The DMA includes a controller coupled to the sensor and configured to determine a first set of estimated radial distances to an object in the local area based on the one or more TOF phase shifts. The controller determines a second estimated radial distance to the object based on an encoding of structured light and at least one of the captured images. The controller selects an estimated radial distance from the first set of radial distances.

Depth measurement assembly with a structured light source and a time of flight camera

Growing community of inventors

Bellevue, WA, United States of America

Michael Anthony Hall