Meta Platforms Technologies, LLC

Reid Pinkham

Andrew Samuel Berkovich

Ziyun Li

Shlomo Alkalay

Jack Thomas Lavier

Tanner Schmidt

Systems and readout methods for foveated sensing may include a pixel array, readout circuitry, and processing logic. The pixel array may have a number of pixels. The readout circuitry may be configured to read image data from the pixel array for each of the plurality of pixels. The processing logic may be configured to identify a number of regions of interest (ROIs) within the pixel array. The processing logic may be configured to associate the image data for the pixels with one or more ROIs. The processing logic may be configured to arrange the image data into data frames. The data frames may include the image data ordered by ROI. Image data for inactive pixels may be removed from the image data and data frames prior to transmission. The processing logic may be configured to transmit the data frames in an order that is based on the ROIs.

Readout methods for foveated sensing

Features described herein pertain to low latency hierarchical image capture. An image can be captured using a pixel array of an image sensing system. A region-of-interest (ROI) can be detected in the image. Image characteristics of the image can be determined based on the region-of-interest (ROI). Determining the image characteristics of the image can include determining an image quality level of the region-of-interest (ROI). Image capturing instructions for capturing a set of images can be determined based on the image characteristics. The set of images can be combined into an image and an object can be recognized in the image.

Low latency hierarchical image capture

Various embodiments set forth computer vision systems. In some embodiments, one or more activation maps are generated based on eye gaze data and used to control operation of one or more sensors. Each activation map indicates a highest density for acquiring sensor data in a region corresponding to an eye gaze direction and lower densities for acquiring sensor data in other regions. After the sensor data is acquired, a processing pipeline sub-samples (or bins) and processes regions of the acquired sensor data at different ratios based on the eye gaze direction. The sub-sampled (or binned) data for each region is processed independently, and results of the processing are combined to generate processed data, such as an image that can be displayed to a user.

Techniques for gaze-contingent sensing and processing

In one example, an apparatus comprises a pixel cell array and a controller formed within a semiconductor package. The pixel cell array is configured to: generate, at a first time and based on first programming signals received from the controller, a first image frame; transmit the first image frame to a host processor; and transmit the first image frame or a second image frame to the controller, the second image frame being generated at the first time and having a different sparsity of pixels from the first image frame. The controller is configured to receive second programming signals from a host processor, the second programming signals being determined by the host processor based on the first image frame; update the first programming signals based on the second programming signals; and control the pixel cell array to generate a subsequent image frame based on the updated first programming signals.

Dynamically programmable image sensor

In one example, an apparatus comprises: an image sensor comprising a plurality of pixel cells; a frame buffer; and a sensor compute circuit configured to: receive, from the frame buffer, a first image frame comprising first active pixels and first inactive pixels, the first active pixels being generated by a first subset of the pixel cells selected based on first programming data; perform an image-processing operation on a first subset of pixels of the first image frame, whereby a second subset of pixels of the first image frame are excluded from the image-processing operation, to generate a processing output; based on the processing output, generate second programming data; and transmit the second programming data to the image sensor to select a second subset of the pixel cells to generate second active pixels for a second image frame.

Growing community of inventors

Redmond, WA, United States of America

Reid Pinkham