Google Inc.

Synopsys, Inc.

Rahul Garg

Neal Wadhwa

Marc Stewart Levoy

Jiawen Chen

Yael Pritch Knaan

Shiqi Chen

Steven Maxwell Seitz

Jonathan Tilton Barron

Jonathan Tompson

Chee Heng Chew

Christian Plagemann

Navneet Dalal

Antonio Bernardo Monteiro Costa

Mark Chang

Thor Carpenter

Mehul Nariyawala

Sameer Ansari

Irena Kemelmaher

Pratul Preeti Srinivasan

Manuel Christian Clement

Alexander James Faaborg

Johnny Hsienchow Lee

Shahram Izadi

Joel Hesch

Dominik Philemon Kaeser

Tianfan Xue

Gregory Marshall Burgess

Parijat Biswas

Sean Ryan Fanello

Timothy Samuel Psiaki

Yinda Zhang

Yicheng Wu

Christian Haene

David Edward Jacobs

Sitikant Sahu

Qiurui He

Sergio Orts Escolano

Tim Psiaki

Greg Burgess

Shumian Xin

Example implementations relate to joint depth prediction from dual cameras and dual pixels. An example method may involve obtaining a first set of depth information representing a scene from a first source and a second set of depth information representing the scene from a second source. The method may further involve determining, using a neural network, a joint depth map that conveys respective depths for elements in the scene. The neural network may determine the joint depth map based on a combination of the first set of depth information and the second set of depth information. In addition, the method may involve modifying an image representing the scene based on the joint depth map. For example, background portions of the image may be partially blurred based on the joint depth map.

Joint depth prediction from dual-cameras and dual-pixels

A method includes obtaining split-pixel image data including a first sub-image and a second sub-image. The method also includes determining, for each respective pixel of the split-pixel image data, a corresponding position of a scene feature represented by the respective pixel relative to a depth of field, and identifying, based on the corresponding positions, out-of-focus pixels. The method additionally includes determining, for each respective out-of-focus pixel, a corresponding pixel value based on the corresponding position, a location of the respective out-of-focus pixel within the split-pixel image data, and at least one of: a first value of a corresponding first pixel in the first sub-image or a second value of a corresponding second pixel in the second sub-image. The method further includes generating, based on the corresponding pixel values, an enhanced image having an extended depth of field.

Merging split-pixel data for deeper depth of field

A method includes obtaining an input image that contains a particular representation of lens flare, and processing the input image by a machine learning model to generate a de-flared image that includes the input image with at least part of the particular representation of lens flare removed. The machine learning (ML) model may be trained by generating training images that combine respective baseline images with corresponding lens flare images. For each respective training image, a modified image may be determined by processing the respective training image by the ML model, and a loss value may be determined based on a loss function comparing the modified image to a corresponding baseline image used to generate the respective training image. Parameters of the ML model may be adjusted based on the loss value determined for each respective training image and the loss function.

Learning-based lens flare removal

A method includes obtaining dual-pixel image data that includes a first sub-image and a second sub-image, and generating an in-focus image, a first kernel corresponding to the first sub-image, and a second kernel corresponding to the second sub-image. A loss value may be determined using a loss function that determines a difference between (i) a convolution of the first sub-image with the second kernel and (ii) a convolution of the second sub-image with the first kernel, and/or a sum of (i) a difference between the first sub-image and a convolution of the in-focus image with the first kernel and (ii) a difference between the second sub-image and a convolution of the in-focus image with the second kernel. Based on the loss value and the loss function, the in-focus image, the first kernel, and/or the second kernel, may be updated and displayed.

Defocus blur removal and depth estimation using dual-pixel image data

Apparatus and methods related to using machine learning to determine depth maps for dual pixel images of objects are provided. A computing device can receive a dual pixel image of at least a foreground object. The dual pixel image can include a plurality of dual pixels. A dual pixel of the plurality of dual pixels can include a left-side pixel and a right-side pixel that both represent light incident on a single dual pixel element used to capture the dual pixel image. The computing device can be used to train a machine learning system to determine a depth map associated with the dual pixel image. The computing device can provide the trained machine learning system.

Depth prediction from dual pixel images

Hardware simulation systems and methods for reducing signal dumping time and size of by fast dynamical partial aliasing of signals having similar waveform are provided. One example system is configured to receive, in real-time, a first signal from a producer entity; determine a first signal signature associated with the first signal; determine, in real-time, a second signal signature associated with the second signal; upon determining that the first signal signature matches the second signal signature, designate the first signal as a master signal and designate the second signal as a slave signal; and stop dumping the second signal to a storage space.

Hardware simulation systems and methods for reducing signal dumping time and size by fast dynamical partial aliasing of signals having similar waveform

A camera may capture an image of a scene and use the image to generate a first and a second subpixel image of the scene. The pair of subpixel images may be represented by a first set of subpixels and a second set of subpixels from the image respectively. Each pixel of the image may include two green subpixels that are respectively represented in the first and second subpixel images. The camera may determine a disparity between a portion of the scene as represented by the pair of subpixel images and may estimate a depth map of the scene that indicates a depth of the portion relative to other portions of the scene based on the disparity and a baseline distance between the two green subpixels. A new version of the image may be generated with a focus upon the portion and with the other portions of the scene blurred.

Estimating depth using a single camera

Systems and methods for context-sensitive hand interaction with an immersive environment are provided. An example method includes determining a contextual factor for a user and selecting an interaction mode based on the contextual factor. The example method may also include monitoring a hand of the user to determine a hand property and determining an interaction with an immersive environment based on the interaction mode and the hand property.

Context-sensitive hand interaction

Example embodiments allow for training of artificial neural networks (ANNs) to generate depth maps based on images. The ANNs are trained based on a plurality of sets of images, where each set of images represents a single scene and the images in such a set of images differ with respect to image aperture and/or focal distance. An untrained ANN generates a depth map based on one or more images in a set of images. This depth map is used to generate, using the image(s) in the set, a predicted image that corresponds, with respect to image aperture and/or focal distance, to one of the images in the set. Differences between the predicted image and the corresponding image are used to update the ANN. ANNs tramed in this manner are especially suited for generating depth maps used to perform simulated image blur on small-aperture images.

Aperture supervision for single-view depth prediction

A method for controller tracking with multiple degrees of freedom includes generating depth data at an electronic device based on a local environment proximate the electronic device. A set of positional data is generated for at least one spatial feature associated with a controller based on a pose of the electronic device, as determined using the depth data, relative to the at least one spatial feature associated with the controller. A set of rotational data is received that represents three degrees-of-freedom (3DoF) orientation of the controller within the local environment, and a six degrees-of-freedom (6DoF) position of the controller within the local environment is tracked based on the set of positional data and the set of rotational data.

Controller tracking for multiple degrees of freedom

In one aspect, a method and system are described for receiving input for a virtual user in a virtual environment. The input may be based on a plurality of movements performed by a user accessing the virtual environment. Based on the plurality of movements, the method and system can include detecting that at least one portion of the virtual user is within a threshold distance of a collision zone, the collision zone being associated with at least one virtual object. The method and system can also include selecting a collision mode for the virtual user based on the at least one portion and the at least one virtual object and dynamically modifying the virtual user based on the selected collision mode.

Context sensitive hand collisions in virtual reality

Systems and methods are disclosed for gesture-initiated actions in videoconferences. In one implementation, a processing device receives content streams during a communication session, identifies a request for feedback within one of the content streams, based on an identification of the request for feedback, processes the content streams to identify one or more gestures within at least one of the content streams, and based on a determination that a first gesture of the one or more gestures is relatively more prevalent across the content streams than one or more other gestures, initiates an action with respect to the communication session.

Gesture-initiated actions in videoconferences

Systems and methods are disclosed for gesture-initiated actions in videoconferences. In one implementation, a processing device receives one or more content streams as part of a communication session. The processing device identifies, within the one or more content streams, a request for feedback. The processing device processes, based on an identification of a request for feedback within the one of the plurality of content streams, the one or more content streams to identify a presence of one or more gestures within at least one of the one or more content streams. The processing device initiates, based on an identification of the presence of one or more gestures within at least one of the one or more content streams, an action with respect to the communication session.

Growing community of inventors

Sunnyvale, CA, United States of America

Rahul Garg