Google Inc.

Bryan Andrew Seybold

Susanna Maria Ricco

Sudheendra Vijayanarasimhan

Rahul Sukthankar

Sourish Chaudhuri

Cordelia Luise Schmid

Aikaterini Fragkiadaki

Achal D Dave

A system comprising an encoder neural network, a scene structure decoder neural network, and a motion decoder neural network. The encoder neural network is configured to: receive a first image and a second image; and process the first image and the second image to generate an encoded representation of the first image and the second image. The scene structure decoder neural network is configured to process the encoded representation to generate a structure output characterizing a structure of a scene depicted in the first image. The motion decoder neural network configured to process the encoded representation to generate a motion output characterizing motion between the first image and the second image.

Determining structure and motion in images using neural networks

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for training an optical flow object localization system and a novel object localization system. In a first aspect, the optical flow object localization system is trained to process an optical flow image to generate object localization data defining locations of objects depicted in a video frame corresponding to the optical flow image. In a second aspect, a novel object localization system is trained to process a video frame to generate object localization data defining locations of novel objects depicted in the video frame.

Processing images to localize novel objects

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for audio classifiers. In one aspect, a method includes obtaining a plurality of video frames from a plurality of videos, wherein each of the plurality of video frames is associated with one or more image labels of a plurality of image labels determined based on image recognition; obtaining a plurality of audio segments corresponding to the plurality of video frames, wherein each audio segment has a specified duration relative to the corresponding video frame; and generating an audio classifier trained using the plurality of audio segment and the associated image labels as input, wherein the audio classifier is trained such that the one or more groups of audio segments are determined to be associated with respective one or more audio labels.

Audio classifier

A temporal difference model can be trained to receive at least a first state representation and a second state representation that respectively describe a state of an object at two different times and, in response, output a temporal difference representation that encodes changes in the object between the two different times. To train the model, the temporal difference model can be combined with a prediction model that, given the temporal difference representation and the first state representation, seeks to predict or otherwise reconstruct the second state representation. The temporal difference model can be trained on a loss value that represents a difference between the second state representation and the prediction of the second state representation. In such fashion, unlabeled data can be used to train the temporal difference model to provide a temporal difference representation. The present disclosure further provides example uses for such temporal difference models once trained.

Growing community of inventors

San Francisco, CA, United States of America

Bryan Andrew Seybold