Samsung Electronics Co., Ltd.

Google Inc.

Changgeng Liu

Ernest Rehmatulla Post

Ye Zhao

Brian R Patton

Kushal Kardam Vyas

Congzhong Guo

Sajid Hassan Sadi

Ion Bita

Marek Mienko

Abdelrahman Abdelhamed

Abhijith Punnappurath

Luxi Zhao

Gustavo A Guayaquil Sosa

Xi Chen

Ziwen Jiang

Kishore Rathinavel

In one embodiment, a method includes generating, from an accessed image, one or more image patches, and for each image patch: (1) accessing a set of point-spread functions (PSFs), wherein each PSF in the set of PSFs corresponds to one of a plurality of points in the image patch; (2) generating a set of deconvolved image patches by deconvolving the image patch with each PSF from the set of PSFs for that patch; (3) determining, for each of one or more portions of the image patch, a set of weights, wherein each weight in the set of weights is associated with one of the deconvolved image patches; and (4) generating a restored image patch by interpolating the set of deconvolved image patches based on the set of weights.

Restoring images using deconvolution

A method includes accessing a display structure for manufacturing a display panel for an under-display camera (UDC) system. The method further includes computing a point spread function (PSF) of the UDC system with the display structure and computing a metric to evaluate performance of the display structure based on the PSF. The method further includes determining that the metric does not satisfy selection criteria for using the display structure for manufacturing the display panel for the UDC system and in response to the determination, iteratively optimizing the display structure, using an automated searching process, until the metric satisfies the selection criteria. The method further includes generating an optimized display structure responsive to completion of the automated searching process. The optimized display structure is used for manufacturing the display panel for the UDC system.

Automating search for improved display structure for under-display camera systems

This document describes systems and techniques directed at mitigating display diffraction flares for under-display sensing. In aspects, an equation may be derived that models the effects of a display in producing a diffraction phenomenon at an image plane of a sensing region for an under-display light-sensing device. The equation may be used to determine an arrangement (e.g., an optimized arrangement) of components (e.g., sub-pixels) within the display that minimizes a diffraction efficiency for at least one diffraction order and, thereby, mitigates an intensity and/or a prevalence of optical artifacts in light-sensing data. In implementations, an image intensity point-spread-function is utilized to calculate diffraction efficiencies for respective diffraction orders (e.g., the lowest diffraction orders, the diffraction orders with the greatest brightness).

Mitigating display diffraction flares for under-display sensing

A method includes capturing, by a camera disposed behind a display panel of the electronic device, a plurality of images at a plurality of exposures, respectively. One or more captured images include one or more flare artifacts. The method further includes generating a high dynamic range (HDR) image by fusing the captured images. The method further includes accessing a HDR point spread function (PSF) from a memory of the electronic device. The HDR PSF may be generated, at an initial camera calibration stage, by fusing a plurality of PSFs captured at the plurality of exposures, respectively. The method further includes generating, using an image deconvolution technique, a reconstructed image with flare artifacts mitigated based on the HDR image and the HDR PSF.

Flare mitigation via deconvolution using high dynamic range imaging

A method includes capturing, by a camera disposed behind a display panel of an electronic device, an original image through a semi-transparent pixel region of the display panel, and determining a depth position with respect to at least one object identified within the original image. The method further includes accessing, based on the depth position, a plurality of point spread functions (PSFs) corresponding to a plurality of lateral positions at the depth position, and generating a set of image patches based on the plurality of PSFs. Each image patch of the set of image patches is generated based on a different one of the plurality of PSFs. The method concludes with generating a reconstructed image corresponding to the original image based on the set of image patches.

Multiple point spread function based image reconstruction for a camera behind a display

A method includes capturing, by a camera disposed behind a display panel of an electronic device, an original image through a semi-transparent pixel region of the display panel. The original image is associated with one or more predetermined point spread functions (PSFs) and includes one or more blurring artifacts. The method further includes estimating an optimal regularization parameter for the original image based on the one or more predetermined PSFs and the one or more blurring artifacts, applying a self-regularizing inverse filter, such as a fast self-regularizing inverse filter, to the original image based on the estimated regularization parameter, and generating a deblurred image based on the self-regularizing inverse filter. Generating the deblurred image includes reconstructing the original image utilizing the self-regularizing inverse filter to remove the one or more blurring artifacts.

Self-regularizing inverse filter for image deblurring

A method includes capturing, by a camera disposed behind a display panel of an electronic device, a plurality of point spread functions (PSFs) through a semi-transparent pixel region of the display panel. Each of the plurality of PSFs is captured at a different exposure time. The method further includes generating an intensity dataset corresponding to the plurality of PSFs, the intensity dataset comprises a plurality of pixel location and a plurality of exposure times associated with the respective pixel locations, calculating a plurality of noise statistics values for the plurality of pixel locations of the intensity dataset, respectively, generating a pixel mask, the pixel mask filters the plurality of pixel locations for pixel locations with respective noise statistics values within a particular threshold, generating one or more high dynamic range (HDR) PSFs utilizing the plurality of PSFs and the pixel mask.

High dynamic range point spread function generation for image reconstruction

A method includes capturing, by a camera disposed behind a display panel of an electronic device, an original image through a semi-transparent pixel region of the display panel. The original image includes one or more color components. The method further includes determining, for a plurality of pixel regions of the original image, a point spread function (PSF) for each of the one or more color components. The method further includes performing, for the plurality of pixel regions of the original image, a deconvolution of each of the one or more color components of the original image based at least in part on their respective PSFs. The method thus includes generating a reconstructed image corresponding to the original image based on the deconvolutions of the one or more color components of the plurality of pixel regions of the original image.

Processing images captured by a camera behind a display

A method includes capturing light reflected off one or more objects and splitting the captured light into a plurality of light fields with a plurality of phase curvatures. The method further includes generating, for the one or more objects, a plurality of holograms based on the plurality of light fields and determining, for each of the plurality of holograms, an intensity and a phase-shift of the hologram. The method thus includes generating a depth map comprising depth information for the one or more objects based at least in part on a function determined by the respective phase-shifts of the plurality of holograms.

Incoherent digital holography based depth camera

A method includes capturing, by a camera disposed behind a display panel of an electronic device, a plurality of point spread functions (PSFs) through a semi-transparent pixel region of the display panel. Each of the plurality of PSFs is captured at a different exposure time. The method further includes determining, for each of the PSFs, pixel intensity data for each of a plurality of pixel locations of the PSF. The pixel intensity data is associated with the exposure time of the respective PSF. The method further includes calculating, for each pixel location, a weighted average pixel intensity value based on the pixel intensity data and the exposure time for the respective pixel location over the plurality of PSFs, and generating a high dynamic range (HDR) PSF utilizing the weighted average pixel intensity values.

Growing community of inventors

San Jose, CA, United States of America

Changgeng Liu