Facebook Technologies, LLC

Meta Platforms Technologies, LLC

Bae Systems Imaging Solutions Inc.

Fairchild Imaging, Inc.

Leland Stanford Junior University

Canesta, Inc.

Microsoft Technology Licensing, LLC

Oculus, Vr, LLC

Xinqiao Liu

Song Chen

Andrew Samuel Berkovich

Michael Anthony Hall

Boyd Fowler

Qing Chao

Huixuan Tang

Zhaoming Zhu

Shuochen Su

Tsung-Hsun Tsai

Wei Gao

Rajesh Lachhmandas Chhabria

Zihe Gao

Hans Reyserhove

Nicholas Daniel Trail

Cyrus Soli Bamji

Richard Andrew Newcombe

Byron Taylor

Ziyun Li

Steven Onishi

Lu Lu

Abbas A El Gamal

Yijing Fu

Steven John Lovegrove

David D Wen

Barbara De Salvo

Syed Shakib Sarwar

Renzo De Nardi

Hung T Do

Hao Yu

Fengqiang Li

Michael Patrick Schaub

Abbas El Gamal

Jakob Julian Engel

Ahn N Vu

Scott Charles McEldowney

Pasi Saarikko

Suk Hwan Lim

Wanli Chi

Robert Dale Cavin

Hee Yoon Lee

Abbas Rafii

Salih Burak Gokturk

Jeffrey M DiCarlo

Maxwell Parsons

Junren Wang

Lyle David Bainbridge

Carlo Tomasi

Hakan Yalcin

Brian A Wandell

Julian Straub

Mohammadhossein Daraeihajitooei

Ender Tunc Eroglu

Sascha Hallstein

Elias Daniel Guestrin

Asif Khan

Guillaume Lestoquoy

Raffaele Capoccia

Feng Xiao

Anh N Vu

Charles J Arduini

Sukhwan Lim

Randi Cabezas

Chengde Wan

Lu Jie

Michael Hall

An artificial reality system performs local user adaptation of machine learning models and global improvement of the machine learning models while ensuring data security and privacy. An artificial reality headset uses smart sensors and a system on chip (SoC). The smart sensor extracts features from sensor data, encrypts them and transmits the encrypted features to the SoC of the headset. The SoC provides the encrypted features to a server, for example, a server in a cloud platform. The server updates a base model with collected features and sends the updated base model parameters to headsets. The smart sensors of a headset receiving the updated model parameters, balance the model parameters with locally stored feature data, for example, user calibration features to perform user customization of the machine learning model.

Federated on-sensor learning for local device adaptation and global improvement of machine learning model

A sensor assembly for determining one or more features of a local area is presented herein. The sensor assembly includes a plurality of stacked sensor layers. A first sensor layer of the plurality of stacked sensor layers located on top of the sensor assembly includes an array of pixels. The top sensor layer can be configured to capture one or more images of light reflected from one or more objects in the local area. The sensor assembly further includes one or more sensor layers located beneath the top sensor layer. The one or more sensor layers can be configured to process data related to the captured one or more images. A plurality of sensor assemblies can be integrated into an artificial reality system, e.g., a head-mounted display.

Sensor system based on stacked sensor layers

An eye-tracking system includes a substrate transparent to visible light and configured to be placed in a field of view of an eye of a user, a plurality of waveguides on the substrate, a light source optically coupled to the plurality of waveguides, and a plurality of polarization volume holograms (PVHs) in the field of view of the eye of the user. Each PVH of the plurality of PVHs is optically coupled to a respective waveguide of the plurality of waveguides and is configured to couple a respective light beam out of the respective waveguide towards the eye of the user.

Liquid crystal polarization hologram (LCPH) based eye tracking for AR/VR

A depth camera assembly (DCA) optimizes illumination and image capture of a local area to generate depth information of the local area. The DCA determines depth information for a first portion of the local area viewable at a first pose. The DCA is moved from the first pose to a second pose, where a second portion of the local area is viewable and overlaps with the first portion. The overlapping region is not illuminated by the DCA. A non-overlapping portion of the second portion is illuminated, captured, and depth information determined.

Dynamic illumination control for depth determination

In some examples, a sensor apparatus comprises: an array of pixel cells each including one or more photodiodes configured to generate a charge in response to light, and a charge storage device to convert the charge to output a voltage of an array of voltages, one or more an analog-to-digital converter (ADC) configured the convert the array of voltages to first pixel data, and an on-sensor controller configured to input the first pixel data into a machine-learning model to generate output data comprising prediction data associated with one or more features of the first pixel data, generate, based on the prediction data, second pixel data, the second pixel data associated with one or more transformed features of the first pixel data, and send, from the sensor apparatus to a separate receiving apparatus, the second pixel data.

On-sensor image processor utilizing contextual data

Imaging systems, cameras, and image sensors of this disclosure include imaging pixels that include subpixels. Diffractive optical elements such as a metasurface lens layers or a liquid crystal polarization hologram (LCPH) are configured to focus image light to the subpixels of the imaging pixels.

Sensing with liquid crystal polarization holograms and metasurface

An eyebox region is illuminated with a fringe illumination pattern. An event sensor is configured to generate event-signals. Eye motion is determined from the event-signals. Eye-features are extracted from data generated by the event sensors and a predicted gaze vector is generated from the eye-features.

Event sensing and tracking

Methods and systems for performing light measurement are disclosed. In one example, an apparatus comprises an array of pixel cells, each pixel cell of the array of pixel cells configured to perform a light measurement operation and to generate a digital output of the light measurement operation. The apparatus further includes a peripheral circuit configured to: receive a pixel array programming map including programming data targeted at each pixel cell of the array of pixel cells, and configure the light measurement operation at the each pixel cell based on the programming data targeted at the each pixel cell. The apparatus further includes an image processor configured to generate an image frame based on the digital outputs of at least some of the array of pixel cells.

Programmable pixel array

One example described herein includes the digital image sensor having a pixel cell with a photodiode and a quantizer circuit coupled to the pixel cell. The quantizer circuit includes a charge storage device that generates a voltage based on electric charge from the photodiode. The quantizer circuit also includes a single-input comparator that can switch from a first output state to a second output state in response to a ramp signal provided by a ramp generator. The quantizer circuit further includes a memory switch that can cause a counter value from a digital counter to be stored in a digital memory in response to the single-input comparator switching from the first output state to the second output state. The counter value can serve as a digital pixel value associated with the pixel cell.

Digital image sensor using a single-input comparator based quantizer

In one example, a method comprises: exposing a first photodiode to incident light to generate first charge; exposing a second photodiode to the incident light to generate second charge; converting, by a first charge sensing unit, the first charge to a first voltage; converting, by a second charge sensing unit, the second charge to a second voltage; controlling an ADC to detect, based on the first voltage, that a quantity of the first charge reaches a saturation threshold, and to measure a saturation time when the quantity of the first charge reaches the saturation threshold; stopping the exposure of the first photodiode and the second photodiode to the incident light based on detecting that the quantity of the first charge reaches the saturation threshold; and controlling the ADC to measure, based on the second voltage, a quantity of the second charge generated by the second photodiode before the exposure ends.

Pixel sensor having adaptive exposure time

In one example, a mobile device comprises: a physical link; a plurality of image sensors, each image sensor being configured to transmit image data via the physical link; and a controller coupled to the physical link, whereby the physical link, the plurality of image sensors, and the controller form a multi-drop network. The controller is configured to: transmit a control signal to configure image sensing operations at the plurality of image sensors; receive, via the physical link, image data from at least a subset of the plurality of image sensors; combine the image data from the at least a subset of the plurality of image sensors to obtain an extended field of view (FOV); determine information of a surrounding environment of the mobile device captured within the extended FOV; and provide the information to an application to generate content based on the information.

Distributed sensor system

In one example, an apparatus includes an array of pixel cells and a peripheral circuit. Each pixel cell of the array of pixel cells includes: a memory to store pixel-level programming data and measurement data, and light measurement circuits configured to, based on the pixel-level programming data from the control memory, perform a light measurement operation to generate the measurement data, and to store the measurement data at the data memory. The peripheral circuit is configured to: receive a pixel array programming map including pixel-level programming data targeted at each pixel cell; extract the pixel-level programming data from the pixel array programming map; and transmit the pixel-level programming data to each pixel cell of the array of pixel cells for storage at the memory of the each pixel cell and to individually control the light measurement operation and a generation of the measurement data at the each pixel cell.

In some examples, a sensor apparatus comprises: a pixel cell configured to generate a voltages, the pixel cell including a photodiode configured to generate charge in response to incoming light, and a charge storage device to convert the charge to a voltage; an integrated circuit configured to: determine a first captured voltage converted by the charge storage device during a first time period; compare the first captured voltage to a threshold voltage value; and in response to determining that the first captured voltage meets or exceeds the threshold voltage value: determine first time data corresponding to the first time period; and prevent the charge storage device from further generating a charge; and an analog-to-digital converter (ADC) configured to generate a digital pixel value based on the first captured voltage, and a memory to store the digital pixel value and the first time data.

Digital pixel sensor having reduced quantization operation

In one example, an apparatus comprises: a first photodiode to generate a first charge; a second photodiode to generate a second charge; a quantizer; a first memory bank and a second memory bank; and a controller configured to: control the quantizer to perform a first quantization operation and a second quantization operation of the first charge to generate, respectively, a first digital output and a second digital output, the first and second quantization operations being associated with different intensity ranges; store one of the first digital output or the second digital output in the first memory bank; control the quantizer to perform a third quantization operation of the second charge to generate a third digital output, the third quantization operation being associated with a different intensity range from at least one of the first or second quantization operations; and store the third digital output in the second memory bank.

Pixel sensor having multiple photodiodes and shared comparator

In one example, an apparatus comprises: a photodiode to generate a charge in response to light within an exposure period having a first duration; a charge sensing unit to accumulate at least a part of the charge within the exposure period; a quantizer; and a controller to: determine, using the quantizer and within a measurement period having a second duration, whether a first quantity of the at least a part of the charge accumulated at the charge sensing unit exceeds a threshold, and a time it takes for the first quantity to exceed the threshold, wherein the first duration and the second duration are individually programmable; and based on whether the first quantity exceeds the threshold, output a first value representing the time or a second value representing a second quantity of the charge generated by the photodiode within the exposure period to represent an intensity of the light.

Non-linear quantization at pixel sensor

Growing community of inventors

Medina, WA, United States of America

Xinqiao Liu