Google Inc.

Waymo LLC

Kevin Patrick Murphy

Yu He

Aditee Kumthekar

Jonathon Shlens

Chen Sun

Abhinav Shrivastava

Sergio Guadarrama Cotado

Carl Martin Vondrick

Wei Zhang

Jonathan J Huang

Congcong Li

David A Ross

Barret Zoph

Alper Ayvaci

Ruichi Yu

Rahul Sukthankar

Andrew R Golding

Wei Hua

Mohammad Norouzi

Junhua Mao

Evgeniy Gabrilovich

Caroline Rebecca Pantofaru

Jia Li

Cordelia Luise Schmid

Konstantinos Bousmalis

Susanna Maria Ricco

David J Bieber

Fei-Fei Li

Stephan Gouws

Jeffrey Jon Dalton

Karthik Raman

Frederick Bertsch

Ryan Lienhart Dahl

Amelie Royer

Alireza Fathi

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for implementing a machine-learned object tracking policy. One of the methods includes receiving a current video frame by a user device having a plurality of installed object trackers, wherein each object tracker is configured to perform a different object tracking procedure on the current video frame rent video frame. The current video frame and one or more object tracks previously generated by the one or more object trackers are provided as input to a trained policy engine that implements a reinforcement learning model to generate a particular object tracking plan. A particular object tracking plan is selected based on the output of the reinforcement learning model, and the selected object tracking plan is performed on the current video frame to generate one or more updated object tracks for the current video frame.

Adaptive object tracking policy

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for semantically-consistent image style transfer. One of the methods includes: receiving an input source domain image; processing the source domain image using one or more source domain low-level encoder neural network layers to generate a low-level representation; processing the low-level representation using one more high-level encoder neural network layers to generate an embedding of the input source domain image; processing the embedding using one or more high-level decoder neural network layers to generate a high-level feature representation of features of the input source domain image; and processing the high-level feature representation of the features of the input source domain image using one or more target domain low-level decoder neural network layers to generate an output target domain image that is from the target domain but that has similar semantics to the input source domain image.

Semantically-consistent image style transfer

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for performing visual tracking. In one aspect, a method comprises receiving: (i) one or more reference video frames, (ii) respective reference labels for each of a plurality of reference pixels in the reference video frames, and (iii) a target video frame. The reference video frames and the target video frame are processed using a colorization machine learning model to generate respective pixel similarity measures between each of (i) a plurality of target pixels in the target video frame, and (ii) the reference pixels in the reference video frames. A respective target label is determined for each target pixel in the target video frame, comprising: combining (i) the reference labels for the reference pixels in the reference video frames, and (ii) the pixel similarity measures.

Visual tracking by colorization

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for performing action localization in images and videos. In one aspect, a system comprises a data processing apparatus; a memory in data communication with the data processing apparatus and storing instructions that cause the data processing apparatus to perform image processing and video processing operations comprising: receiving an input comprising an image depicting a person; identifying a plurality of context positions from the image; determining respective feature representations of each of the context positions; providing a feature representation of the person and the feature representations of each of the context positions to a context neural network to obtain relational features, wherein the relational features represent relationships between the person and the context positions; and determining an action performed by the person using the feature representation of the person and the relational features.

Action localization in images and videos using relational features

A method for determining an architecture for a task neural network configured to perform a particular machine learning task is described. The method includes obtaining data specifying a current set of candidate architectures for the task neural network; for each candidate architecture in the current set: processing the data specifying the candidate architecture using a performance prediction neural network having multiple performance prediction parameters, the performance prediction neural network being configured to process the data specifying the candidate architecture in accordance with current values of the performance prediction parameters to generate a performance prediction that characterizes how well a neural network having the candidate architecture would perform after being trained on the particular machine learning task; and generating an updated set of candidate architectures by selecting one or more of the candidate architectures in the current set based on the performance predictions for the candidate architectures in the current set.

Neural architecture search using a performance prediction neural network

Systems and methods for transforming grayscale images into color images using deep neural networks are described. One of the systems include one or more computers and one or more storage devices storing instructions that, when executed by one or more computers, cause the one or more computers to implement a coloring neural network, a refinement neural network, and a subsystem. The coloring neural network is configured to receive a first grayscale image having a first resolution and to process the first grayscale image to generate a first color image having a second resolution lower than the first resolution. The subsystem processes the first color image to generate a set of intermediate image outputs. The refinement neural network is configured to receive the set intermediate image outputs, and to process the set of intermediate image outputs to generate a second color image having a third resolution higher than the second resolution.

Transforming grayscale images into color images using deep neural networks

Aspects of the disclosure relate to training and using a model for identifying actions of objects. For instance, LIDAR sensor data frames including an object bounding box corresponding to an object as well as an action label for the bounding box may be received. Each sensor frame is associated with a timestamp and is sequenced with respect to other sensor frames. Each given sensor data frame may be projected into a camera image of the object based on the timestamp associated with the given sensor data frame in order to provide fused data. The model may be trained using the fused data such that the model is configured to, in response to receiving fused data, the model outputs an action label for each object bounding box of the fused data. This output may then be used to control a vehicle in an autonomous driving mode.

Object action classification for autonomous vehicles

Systems and methods provide distantly supervised wrapper induction for semi-structured documents, including automatically generating and annotating training documents for the wrapper. Training of the wrapper may occur in two phases using the training documents. An example method includes identifying a training set of semi-structured web pages having a subject entity that exists in a knowledge base and, for each training page, identifying target objects, identifying predicates in the knowledge base that connect the subject entity to a target objects identified in the training page, and annotating the training page. Annotating a training page includes generating a feature set for a mention of the target object, generating predicate-target object pairs for the mention, and labeling each predicate-target object pair with a corresponding example type and weight. The annotated training pages are used to train the wrapper to extract new subject entities and new facts from the set of semi-structured web pages.

Distantly supervised wrapper induction for semi-structured documents

To generate navigation directions for a driver of a vehicle, a route for guiding the driver to a destination is obtained, visual landmarks corresponding to prominent physical objects disposed along the route are retrieved, and real-time imagery is collected at the vehicle approximately from a vantage point of the driver during navigation along the route. Using (i) the retrieved visual landmarks and (ii) the imagery collected at the vehicle, a subset of the visual landmarks that are currently visible to the driver is selected. Navigation directions describing the route are provided the driver, the navigation directions referencing the selected subset of the visual landmarks and excluding the remaining visual landmarks.

Systems and methods for using real-time imagery in navigation

Mechanisms are provided that: identify topics associated with a plurality of pieces of media content presented in a session; calculate a distance metric for pairs of topics, wherein each of the pairs of topics includes a first topic associated with a first piece of media content and a second topic associated with a second piece of media content, and wherein the second piece of media content was presented within a given span of the presentation of the first piece of media content; for each first topic of the pairs of topics, generate a rank-ordered list for all corresponding second topics; for each of the plurality of pieces of media content, generate a single rank-ordered list of all second topics; and for each of the plurality of pieces of media content, identify one or more other pieces of media content as recommended media content based on the single rank-ordered list.

Methods, systems, and media for recommending media content

Methods, system, and media for recommending media content

Growing community of inventors

Palo Alto, CA, United States of America

Kevin Patrick Murphy