Google Inc.

Waymo LLC

Jonathan Tilton Barron

Yun-Ta Tsai

Jiawen Chen

Tianfan Xue

Ryan M Geiss

Stephen J DiVerdi

Pratul Preeti Srinivasan

Neal Wadhwa

Rahul Garg

Alexey Dosovitskiy

Seyed Mohammad Mehdi Sajjadi

Tiancheng Sun

Daniel Christopher Duckworth

Jian Jim Wang

Sofien Bouaziz

Samuel William Hasinoff

Sean Ryan Francesco Fanello

Christoph Rhemann

Ricardo Martin Brualla

Ricardo Martin-Brualla

Dillon Sharlet

Yun-ta Tsai

Steven Maxwell Seitz

Marc Stewart Levoy

Onur Gonen Guleryuz

Paul E Debevec

Daniel Goldman

Shahram Izadi

Adarsh Prakash Murthy Kowdle

Rohit Kumar Pandey

Vladimir Tankovich

Michael John Schoenberg

Ambrus Csaszar

Chloe LeGendre

Sean Ryan Fanello

Konstantine Nicholas John Tsotsos

Henrik Kretzschmar

Yicheng Wu

Sameh Khamis

Mirko Schmidt

Michael Gharbi

Eric Turner

Tsung-Yi Lin

Vivek Verma

Vincent Michael Casser

David Edward Jacobs

Navin Sarma

Qiurui He

Julien Valentin

Keunhong Park

Ivan Dryanovski

Thomas Allen Funkhouser

Stephen Joseph Diverdi

Mira Leung

Maksym Dzitsiuk

Noha Waheed Ahmed Radwan

Andrea Tagliasacchi

Yen-Chen Lin

Jian Wang

Vittorio Carlo Ferrari

Konstantinos Rematas

Benjamin Joseph Mildenhall

Matthew Justin Tancik

Shumian Xin

Xiuming Zhang

Peter Raymond Florence

Michael Milne

Noha Radwan

Sabeek Mani Pradhan

Benjamin Michael Poole

Jose Pascoal

João Manuel Castro Afonso

Utkarsh Sinha

Xuaner Zhang

Andrew Huaming Liu

Ben Poole

Yun-Ta Tsia

Techniques of image synthesis using a neural radiance field (NeRF) includes generating a deformation model of movement experienced by a subject in a non-rigidly deforming scene. For example, when an image synthesis system uses NeRFs, the system takes as input multiple poses of subjects for training data. In contrast to conventional NeRFs, the technical solution first expresses the positions of the subjects from various perspectives in an observation frame. The technical solution then involves deriving a deformation model, i.e., a mapping between the observation frame and a canonical frame in which the subject's movements are taken into account. This mapping is accomplished using latent deformation codes for each pose that are determined using a multilayer perceptron (MLP). A NeRF is then derived from positions and casted ray directions in the canonical frame using another MLP. New poses for the subject may then be derived using the NeRF.

Deformable neural radiance fields

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for rendering a new image that depicts a scene from a perspective of a camera at a new camera viewpoint.

Large scene neural view synthesis

Systems and methods described herein may relate to potential methods of training a machine learning model to be implemented on a mobile computing device configured to capture, adjust, and/or store image frames. An example method includes supplying a first image frame of a subject in a setting lit within a first lighting environment and supplying a second image frame of the subject lit within a second lighting environment. The method further includes determining a mask. Additionally, the method includes combining the first image frame and the second image frame according to the mask to generate a synthetic image and assigning a score to the synthetic image. The method also includes training a machine learning model based on the assigned score to adjust a captured image based on the synthetic image.

Systems and methods for manipulation of shadows on portrait image frames

Apparatus and methods related to applying lighting models to images of objects are provided. An example method includes applying a geometry model to an input image to determine a surface orientation map indicative of a distribution of lighting on an object based on a surface geometry. The method further includes applying an environmental light estimation model to the input image to determine a direction of synthetic lighting to be applied to the input image. The method also includes applying, based on the surface orientation map and the direction of synthetic lighting, a light energy model to determine a quotient image indicative of an amount of light energy to be applied to each pixel of the input image. The method additionally includes enhancing, based on the quotient image, a portion of the input image. One or more neural networks can be trained to perform one or more of the aforementioned aspects.

Enhanced photo relighting based on machine learning models

Provided are systems and methods that invert a trained NeRF model, which stores the structure of a scene or object, to estimate the 6D pose from an image taken with a novel view. 6D pose estimation has a wide range of applications, including visual localization and object pose estimation for robot manipulation.

Inverting neural radiance fields for pose estimation

Apparatus and methods related to applying lighting models to images of objects are provided. A neural network can be trained to apply a lighting model to an input image. The training of the neural network can utilize confidence learning that is based on light predictions and prediction confidence values associated with lighting of the input image. A computing device can receive an input image of an object and data about a particular lighting model to be applied to the input image. The computing device can determine an output image of the object by using the trained neural network to apply the particular lighting model to the input image of the object.

Photo relighting using deep neural networks and confidence learning

Systems and methods for view synthesis and three-dimensional reconstruction can learn an environment by utilizing a plurality of images of an environment and depth data. The use of depth data can be helpful when the quantity of images and different angles may be limited. For example, large outdoor environments can be difficult to learn due to the size, the varying image exposures, and the limited variance in view direction changes. The systems and methods can leverage a plurality of panoramic images and corresponding lidar data to accurately learn a large outdoor environment to then generate view synthesis outputs and three-dimensional reconstruction outputs. Training may include the use of an exposure correction network to address lighting exposure differences between training images.

Radiance fields for three-dimensional reconstruction and novel view synthesis in large-scale environments

Provided are systems and methods for synthesizing novel views of complex scenes (e.g., outdoor scenes). In some implementations, the systems and methods can include or use machine-learned models that are capable of learning from unstructured and/or unconstrained collections of imagery such as, for example, 'in the wild' photographs. In particular, example implementations of the present disclosure can learn a volumetric scene density and radiance represented by a machine-learned model such as one or more multilayer perceptrons (MLPs).

View synthesis robust to unconstrained image data

Examples relate to implementations of a neural light transport. A computing system may obtain data indicative of a plurality of UV texture maps and a geometry of an object. Each UV texture map depicts the object from a perspective of a plurality of perspectives. The computing system may train a neural network to learn a light transport function using the data. The light transport function may be a continuous function that specifies how light interacts with the object when the object is viewed from the plurality of perspectives. The computing system may generate an output UV texture map that depicts the object from a synthesized perspective based on an application of the light transport function by the trained neural network.

Neural light transport

A method includes obtaining an input image that contains a particular representation of lens flare, and processing the input image by a machine learning model to generate a de-flared image that includes the input image with at least part of the particular representation of lens flare removed. The machine learning (ML) model may be trained by generating training images that combine respective baseline images with corresponding lens flare images. For each respective training image, a modified image may be determined by processing the respective training image by the ML model, and a loss value may be determined based on a loss function comparing the modified image to a corresponding baseline image used to generate the respective training image. Parameters of the ML model may be adjusted based on the loss value determined for each respective training image and the loss function.

Learning-based lens flare removal

A method includes obtaining dual-pixel image data that includes a first sub-image and a second sub-image, and generating an in-focus image, a first kernel corresponding to the first sub-image, and a second kernel corresponding to the second sub-image. A loss value may be determined using a loss function that determines a difference between (i) a convolution of the first sub-image with the second kernel and (ii) a convolution of the second sub-image with the first kernel, and/or a sum of (i) a difference between the first sub-image and a convolution of the in-focus image with the first kernel and (ii) a difference between the second sub-image and a convolution of the in-focus image with the second kernel. Based on the loss value and the loss function, the in-focus image, the first kernel, and/or the second kernel, may be updated and displayed.

Defocus blur removal and depth estimation using dual-pixel image data

Scenes can be imaged under low-light conditions using flash photography. However, the flash can be irritating to individuals being photographed, especially when those individuals' eyes have adapted to the dark. Additionally, portions of images generated using a flash can appear washed-out or otherwise negatively affected by the flash. These issues can be addressed by using a flash at an invisible wavelength, e.g., an infrared and/or ultraviolet flash. At the same time a scene is being imaged, at the invisible wavelength of the invisible flash, the scene can also be imaged at visible wavelengths. This can include simultaneously using both a standard RGB camera and a modified visible-plus-invisible-wavelengths camera (e.g., an 'IR-G-UV' camera). The visible and invisible image data can then be combined to generate an improved visible-light image of the scene, e.g., that approximates a visible light image of the scene, had the scene been illuminated during daytime light conditions.

Dark flash photography with a stereo camera

Growing community of inventors

Alameda, CA, United States of America

Jonathan Tilton Barron