Meta Platforms Technologies, LLC

Ziyun Li

Xinqiao Liu

Syed Shakib Sarwar

Barbara De Salvo

Robert Dale Cavin

Andrew Samuel Berkovich

Qing Chao

Manan Suri

Mohammadhossein Daraeihajitooei

Hans Reyserhove

Sascha Hallstein

Reid Pinkham

Asif Khan

Hsien-Hsin Sean Lee

Guillaume Lestoquoy

Randi Cabezas

Chengde Wan

Vivek Kamalkant Parmar

Raffaele Capoccia

A console and headset system locally trains machine learning models to perform customized online learning tasks. To customize the online learning models for specific users of the system without using outside resources, the system trains the models to compare a target frame to stored calibration frames, rather than directly inferring information about a target frame. During deployment, an embedding is generated for the target frame. A sample embedding that is closest to the target embedding is selected from a group of embeddings of calibration frames. The information about the selected embedding and target embedding and ground truths for the calibration frame are provided as inputs to one of the trained models. The model predicts a difference between the target frame and the calibration frame, which can be used to determine information about the target frame.

Personalized online learning for artificial reality applications

Features described herein pertain to low latency hierarchical image capture. An image can be captured using a pixel array of an image sensing system. A region-of-interest (ROI) can be detected in the image. Image characteristics of the image can be determined based on the region-of-interest (ROI). Determining the image characteristics of the image can include determining an image quality level of the region-of-interest (ROI). Image capturing instructions for capturing a set of images can be determined based on the image characteristics. The set of images can be combined into an image and an object can be recognized in the image.

Low latency hierarchical image capture

An eyebox region is illuminated with a fringe illumination pattern. An event sensor is configured to generate event-signals. Eye motion is determined from the event-signals. Eye-features are extracted from data generated by the event sensors and a predicted gaze vector is generated from the eye-features.

Event sensing and tracking

An example apparatus having a heterogenous memory system includes a first sensor layer, of a plurality of stacked sensor layers, including an array of pixels; and one or more semiconductor layers of the plurality of stacked sensor layers located beneath the first sensor layer, the one or more semiconductor layers configured to process pixel data output by the array of pixels, the one or more semiconductor layers including a first memory to store most significant bits ('MSBs') of data involved in the processing of the pixel data; a second memory to store least significant bits (“LSBs”) of the data; and wherein the first memory has a lower bit error rate (“BER”) than the second memory.

Heterogeneous memory system

The disclosed computer-implemented method may include (1) receiving a present frame of a video stream, the present frame comprising a present depiction of a multi-segment articulated body system, (2) identifying a previous frame of the video stream that comprises a previous depiction of the multi-segment articulated body system, (3) analyzing the present frame and the previous frame to determine whether the multi-segment articulated body system remained substantially rigid between the previous frame and the present frame, and (4) estimating a pose of the multi-segment articulated body system in the present frame using a first pose estimation computation that treats the multi-segment articulated body system as rigid and that is selected in contrast to a second pose estimation computation based on determining that the multi-segment articulated body system remained substantially rigid between the previous frame and the present frame. Various other methods, systems, and computer-readable media are also disclosed.

Systems and methods for hand pose estimation from video

In one example, an apparatus comprises: a first sensor layer, including an array of pixel cells configured to generate pixel data; and one or more semiconductor layers located beneath the first sensor layer with the one or more semiconductor layers being electrically connected to the first sensor layer via interconnects. The one or more semiconductor layers comprises on-chip compute circuits configured to receive the pixel data via the interconnects and process the pixel data, the on-chip compute circuits comprising: a machine learning (ML) model accelerator configured to implement a convolutional neural network (CNN) model to process the pixel data; a first memory to store coefficients of the CNN model and instruction codes; a second memory to store the pixel data of a frame; and a controller configured to execute the codes to control operations of the ML model accelerator, the first memory, and the second memory.

Image sensor having on-chip compute circuit

Growing community of inventors

Redmond, WA, United States of America

Ziyun Li