Google Inc.

Leland Stanford Junior University

Nokia Corporation

Eth Zuerich

Nokia Technologies Oy

Eth Zurich

Marc Stewart Levoy

Mark A Horowitz

Patrick M Hanrahan

Yi-Ren Ng

Ryan M Geiss

Samuel William Hasinoff

Jiawen Chen

Roman Lewkow

Neal Wadhwa

Rahul Garg

Changyin Zhou

Yael Pritch Knaan

Noy Cohen

Kari Pulli

Robert A Proudfoot

Sameer Ansari

Samuel J Yang

Wei-Chao Chen

Logan Grosenick

Michael J Broxton

Karl A Deisseroth

Christopher Richard Uhlik

Markus H Gross

Jonathan Tilton Barron

Markus Gross

Natasha Gelfand

Peyman Milanfar

Shahram Izadi

Chorong Johnston

Joseph Kieran O'Sullivan

Michelle Chen

Sean Ryan Fanello

Yinda Zhang

Tianfan Xue

Christian Haene

Sriram Thirthala

David Edward Jacobs

Alexander Schiffhauer

Remo Ziegler

Rolf Adelsberger

Sergio Orts Escolano

Aaron S Andalman

Ren Ng

Pat Hanrahan

Kelly Tsai

Ryan Burgess Geiss

Ruiduo Yang

Karl A Disseroth

Natalia Gelfand

Example implementations relate to joint depth prediction from dual cameras and dual pixels. An example method may involve obtaining a first set of depth information representing a scene from a first source and a second set of depth information representing the scene from a second source. The method may further involve determining, using a neural network, a joint depth map that conveys respective depths for elements in the scene. The neural network may determine the joint depth map based on a combination of the first set of depth information and the second set of depth information. In addition, the method may involve modifying an image representing the scene based on the joint depth map. For example, background portions of the image may be partially blurred based on the joint depth map.

Joint depth prediction from dual-cameras and dual-pixels

Apparatus and methods related to photography are provided. A computing device can receive an input image. An object detector of the computing device can determine an object region of interest of the input image that is associated with an object detected in the input image. A trained machine learning algorithm can determine an output photographic region of interest for the input image based on the object region of interest and the input image. The machine learning algorithm can be trained to identify an output photographic region of interest that is suitable for use by a photographic function for image generation. The computing device can generate an output related to the output photographic region of interest.

Determining regions of interest for photographic functions

Apparatus and methods related to applying lighting models to images of objects are provided. A neural network can be trained to apply a lighting model to an input image. The training of the neural network can utilize confidence learning that is based on light predictions and prediction confidence values associated with lighting of the input image. A computing device can receive an input image of an object and data about a particular lighting model to be applied to the input image. The computing device can determine an output image of the object by using the trained neural network to apply the particular lighting model to the input image of the object.

Dual exposure control in a camera system

Display screen or portion thereof with a graphical user interface

Apparatus and methods related to using machine learning to determine depth maps for dual pixel images of objects are provided. A computing device can receive a dual pixel image of at least a foreground object. The dual pixel image can include a plurality of dual pixels. A dual pixel of the plurality of dual pixels can include a left-side pixel and a right-side pixel that both represent light incident on a single dual pixel element used to capture the dual pixel image. The computing device can be used to train a machine learning system to determine a depth map associated with the dual pixel image. The computing device can provide the trained machine learning system.

Depth prediction from dual pixel images

Imaging systems can often gather higher quality information about a field of view than the unaided human eye. For example, telescopes may magnify very distant objects, microscopes may magnify very small objects, and high frame-rate cameras may capture fast motion. The present disclosure includes devices and methods that provide real-time vision enhancement without the delay of replaying from storage media. The disclosed devices and methods may include a live view user interface with two or more interactive features or effects that may be controllable in real-time. Specifically, the disclosed devices and methods may include a live view display and image and other information enhancements, which utilize in-line computation and constant control.

Graphic interface for real-time vision enhancement

A camera may capture an image of a scene and use the image to generate a first and a second subpixel image of the scene. The pair of subpixel images may be represented by a first set of subpixels and a second set of subpixels from the image respectively. Each pixel of the image may include two green subpixels that are respectively represented in the first and second subpixel images. The camera may determine a disparity between a portion of the scene as represented by the pair of subpixel images and may estimate a depth map of the scene that indicates a depth of the portion relative to other portions of the scene based on the disparity and a baseline distance between the two green subpixels. A new version of the image may be generated with a focus upon the portion and with the other portions of the scene blurred.

Estimating depth using a single camera

Various aspects of the present disclosure are directed toward optics and imaging. As may be implemented with one or more embodiments, an apparatus includes one or more phase masks that operate with an objective lens and a microlens array to alter a phase characteristic of light travelling in a path from a specimen, through the objective lens and microlens array and to a photosensor array. Using this approach, the specimen can be imaged with spatial resolution characteristics provided via the altered phase characteristic, which can facilitate construction of an image with enhanced resolution.

Light-field microscopy with phase masking

The present disclosure relates to methods and systems that may reduce pixel noise due to defective sensor elements in optical imaging systems. Namely, a camera may capture a burst of images with an image sensor while adjusting a focus distance setting of an optical element. For example, the image burst may be captured during an autofocus process. The plurality of images may be averaged or otherwise merged to provide a single, aggregate image frame. Such an aggregate image frame may appear blurry. In such a scenario, 'hot' pixels, 'dead' pixels, or otherwise defective pixels may be more easily recognized and/or corrected. As an example, a defective pixel may be removed from a target image or otherwise corrected by replacing a value of the defective pixel with an average value of neighboring pixels.

Calibration of defective image sensor elements

Various approaches to imaging involve selecting directional and spatial resolution. According to an example embodiment, images are computed using an imaging arrangement to facilitate selective directional and spatial aspects of the detection and processing of light data. Light passed through a main lens is directed to photosensors via a plurality of microlenses. The separation between the microlenses and photosensors is set to facilitate directional and/or spatial resolution in recorded light data, and facilitating refocusing power and/or image resolution in images computed from the recorded light data. In one implementation, the separation is varied between zero and one focal length of the microlenses to respectively facilitate spatial and directional resolution (with increasing directional resolution, hence refocusing power, as the separation approaches one focal length).

Variable imaging arrangements and methods therefor

Images are detected in a manner that addresses various challenges as discussed herein. As may be consistent with one or more embodiments, aspects are directed to an apparatus having sets of photosensors that detect light rays received at different angles from a specimen via a microlens array, with the light rays detected by each set of photosensors representing an aliased view of the specimen. An output indicative of aliased views of the specimen is provided. Certain embodiments further include a logic circuit that processes the output and generates a deconvolved volume image by combining aliased views of the specimen as detected by the photosensors.

Optics apparatus with detection of light rays received at different angles for output indicative of aliased views

A device may operate a first image-capture system to capture first image data of a scene. While the first image-capture system is capturing the first image data, the device may operate a second image-capture system to determine an updated value for the first image setting, and send an instruction to the first image-capture system that indicates to use the updated value for the first image setting to continue to capture the first image data.

Using a second camera to adjust settings of first camera

Image data is processed to facilitate focusing and/or optical correction. According to an example embodiment of the present invention, an imaging arrangement collects light data corresponding to light passing through a particular focal plane. The light data is collected using an approach that facilitates the determination of the direction from which various portions of the light incident upon a portion of the focal plane emanate from. Using this directional information in connection with value of the light as detected by photosensors, an image represented by the light is selectively focused and/or corrected.

Imaging arrangements and methods therefor

Growing community of inventors

Stanford, CA, United States of America

Marc Stewart Levoy