Google Inc.

Waymo LLC

Jonathan Tilton Barron

Yun-Ta Tsai

Jiawen Chen

Tianfan Xue

Ryan M Geiss

Stephen J DiVerdi

Pratul Preeti Srinivasan

Neal Wadhwa

Rahul Garg

Alexey Dosovitskiy

Seyed Mohammad Mehdi Sajjadi

Daniel Christopher Duckworth

Tiancheng Sun

Jian Jim Wang

Samuel William Hasinoff

Sean Ryan Francesco Fanello

Christoph Rhemann

Ricardo Martin-Brualla

Yun-ta Tsai

Dillon Sharlet

Marc Stewart Levoy

Onur Gonen Guleryuz

Paul E Debevec

Sofien Bouaziz

Shahram Izadi

Adarsh Prakash Murthy Kowdle

Rohit Kumar Pandey

Vladimir Tankovich

Ambrus Csaszar

Michael John Schoenberg

Sean Ryan Fanello

Ricardo Martin Brualla

Konstantine Nicholas John Tsotsos

Henrik Kretzschmar

Yicheng Wu

Sameh Khamis

Chloe LeGendre

Michael Gharbi

Mirko Schmidt

Vivek Verma

Tsung-Yi Lin

Eric Turner

David Edward Jacobs

Vincent Michael Casser

Qiurui He

Navin Sarma

Julien Valentin

Ivan Dryanovski

Thomas Allen Funkhouser

Stephen Joseph Diverdi

Noha Waheed Ahmed Radwan

Mira Leung

Maksym Dzitsiuk

Yen-Chen Lin

Andrea Tagliasacchi

Jian Wang

Vittorio Carlo Ferrari

Sabeek Mani Pradhan

Michael Milne

Peter Raymond Florence

Xiuming Zhang

João Manuel Castro Afonso

Jose Pascoal

Shumian Xin

Matthew Justin Tancik

Benjamin Joseph Mildenhall

Konstantinos Rematas

Benjamin Michael Poole

Andrew Huaming Liu

Noha Radwan

Xuaner Zhang

Ben Poole

Yun-Ta Tsia

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for rendering a new image that depicts a scene from a perspective of a camera at a new camera viewpoint.

Large scene neural view synthesis

Systems and methods described herein may relate to potential methods of training a machine learning model to be implemented on a mobile computing device configured to capture, adjust, and/or store image frames. An example method includes supplying a first image frame of a subject in a setting lit within a first lighting environment and supplying a second image frame of the subject lit within a second lighting environment. The method further includes determining a mask. Additionally, the method includes combining the first image frame and the second image frame according to the mask to generate a synthetic image and assigning a score to the synthetic image. The method also includes training a machine learning model based on the assigned score to adjust a captured image based on the synthetic image.

Systems and methods for manipulation of shadows on portrait image frames

Apparatus and methods related to applying lighting models to images of objects are provided. An example method includes applying a geometry model to an input image to determine a surface orientation map indicative of a distribution of lighting on an object based on a surface geometry. The method further includes applying an environmental light estimation model to the input image to determine a direction of synthetic lighting to be applied to the input image. The method also includes applying, based on the surface orientation map and the direction of synthetic lighting, a light energy model to determine a quotient image indicative of an amount of light energy to be applied to each pixel of the input image. The method additionally includes enhancing, based on the quotient image, a portion of the input image. One or more neural networks can be trained to perform one or more of the aforementioned aspects.

Enhanced photo relighting based on machine learning models

Provided are systems and methods that invert a trained NeRF model, which stores the structure of a scene or object, to estimate the 6D pose from an image taken with a novel view. 6D pose estimation has a wide range of applications, including visual localization and object pose estimation for robot manipulation.

Inverting neural radiance fields for pose estimation

Apparatus and methods related to applying lighting models to images of objects are provided. A neural network can be trained to apply a lighting model to an input image. The training of the neural network can utilize confidence learning that is based on light predictions and prediction confidence values associated with lighting of the input image. A computing device can receive an input image of an object and data about a particular lighting model to be applied to the input image. The computing device can determine an output image of the object by using the trained neural network to apply the particular lighting model to the input image of the object.

Photo relighting using deep neural networks and confidence learning

Systems and methods for view synthesis and three-dimensional reconstruction can learn an environment by utilizing a plurality of images of an environment and depth data. The use of depth data can be helpful when the quantity of images and different angles may be limited. For example, large outdoor environments can be difficult to learn due to the size, the varying image exposures, and the limited variance in view direction changes. The systems and methods can leverage a plurality of panoramic images and corresponding lidar data to accurately learn a large outdoor environment to then generate view synthesis outputs and three-dimensional reconstruction outputs. Training may include the use of an exposure correction network to address lighting exposure differences between training images.

Radiance fields for three-dimensional reconstruction and novel view synthesis in large-scale environments

Provided are systems and methods for synthesizing novel views of complex scenes (e.g., outdoor scenes). In some implementations, the systems and methods can include or use machine-learned models that are capable of learning from unstructured and/or unconstrained collections of imagery such as, for example, 'in the wild' photographs. In particular, example implementations of the present disclosure can learn a volumetric scene density and radiance represented by a machine-learned model such as one or more multilayer perceptrons (MLPs).

View synthesis robust to unconstrained image data

Examples relate to implementations of a neural light transport. A computing system may obtain data indicative of a plurality of UV texture maps and a geometry of an object. Each UV texture map depicts the object from a perspective of a plurality of perspectives. The computing system may train a neural network to learn a light transport function using the data. The light transport function may be a continuous function that specifies how light interacts with the object when the object is viewed from the plurality of perspectives. The computing system may generate an output UV texture map that depicts the object from a synthesized perspective based on an application of the light transport function by the trained neural network.

Neural light transport

A method includes obtaining an input image that contains a particular representation of lens flare, and processing the input image by a machine learning model to generate a de-flared image that includes the input image with at least part of the particular representation of lens flare removed. The machine learning (ML) model may be trained by generating training images that combine respective baseline images with corresponding lens flare images. For each respective training image, a modified image may be determined by processing the respective training image by the ML model, and a loss value may be determined based on a loss function comparing the modified image to a corresponding baseline image used to generate the respective training image. Parameters of the ML model may be adjusted based on the loss value determined for each respective training image and the loss function.

Learning-based lens flare removal

A method includes obtaining dual-pixel image data that includes a first sub-image and a second sub-image, and generating an in-focus image, a first kernel corresponding to the first sub-image, and a second kernel corresponding to the second sub-image. A loss value may be determined using a loss function that determines a difference between (i) a convolution of the first sub-image with the second kernel and (ii) a convolution of the second sub-image with the first kernel, and/or a sum of (i) a difference between the first sub-image and a convolution of the in-focus image with the first kernel and (ii) a difference between the second sub-image and a convolution of the in-focus image with the second kernel. Based on the loss value and the loss function, the in-focus image, the first kernel, and/or the second kernel, may be updated and displayed.

Defocus blur removal and depth estimation using dual-pixel image data

Scenes can be imaged under low-light conditions using flash photography. However, the flash can be irritating to individuals being photographed, especially when those individuals' eyes have adapted to the dark. Additionally, portions of images generated using a flash can appear washed-out or otherwise negatively affected by the flash. These issues can be addressed by using a flash at an invisible wavelength, e.g., an infrared and/or ultraviolet flash. At the same time a scene is being imaged, at the invisible wavelength of the invisible flash, the scene can also be imaged at visible wavelengths. This can include simultaneously using both a standard RGB camera and a modified visible-plus-invisible-wavelengths camera (e.g., an 'IR-G-UV' camera). The visible and invisible image data can then be combined to generate an improved visible-light image of the scene, e.g., that approximates a visible light image of the scene, had the scene been illuminated during daytime light conditions.

Dark flash photography with a stereo camera

A handheld user device includes a monocular camera to capture a feed of images of a local scene and a processor to select, from the feed, a keyframe and perform, for a first image from the feed, stereo matching using the first image, the keyframe, and a relative pose based on a pose associated with the first image and a pose associated with the keyframe to generate a sparse disparity map representing disparities between the first image and the keyframe. The processor further is to determine a dense depth map from the disparity map using a bilateral solver algorithm, and process a viewfinder image generated from a second image of the feed with occlusion rendering based on the depth map to incorporate one or more virtual objects into the viewfinder image to generate an AR viewfinder image. Further, the processor is to provide the AR viewfinder image for display.

Depth from motion for augmented reality for handheld user devices

Growing community of inventors

Alameda, CA, United States of America

Jonathan Tilton Barron