Google Inc.

Waymo LLC

Lemon Inc.

Ayar Labs, Inc.

Chen Sun

Congcong Li

Jiyang Gao

Cordelia Luise Schmid

Yifan Xu

Yi Shen

Hang Zhao

Dragomir Dimitrov Anguelov

Junhua Mao

Kevin Patrick Murphy

Benjamin Sapp

Yuning Chai

Yunzhu Li

Tian Lan

Yue Shen

Sanketh Shetty

Ariel Fuxman

Alper Ayvaci

Ruichi Yu

Guangyao Ni

Engin Cinar Sahin

Rahul Sukthankar

Lin Zhou

Daniel Walevski

Mostafa Dehghani

Zhen Li

Kin Chung Wong

Abhinav Shrivastava

Dong Zhao

Georg Heigold

August Yadon

Jingjing Zhao

Haiying Cheng

WeiShan Yu

Aleksei Timofeev

Mario Lucic

Chenyu Li

Chun-Ta Lu

Gamze Inanc

Masoud Loghmani

Yanli Li

Nick Newberg

Carl Martin Vondrick

Zeng Dai

Anurag Arnab

Atanas Frengov

Ari Shapiro

Seong Kuan Wang

Xingyi Shi

Jian Huang

Richard F Tucker

Haran Talmon

Manan Shah

Chao Jia

Krishnamurthy Viswanathan

The present disclosure describes techniques of generating voices for virtual characters. A plurality of source sounds may be received. The plurality of source sounds may correspond to a plurality of frames of a video. The video may comprise a virtual character. The plurality of source sounds may be converted into a plurality of representations in a latent space using a first model. Each representation among the plurality of representations may comprise a plurality of parameters. The plurality of parameters may correspond to a plurality of sound features. A plurality of sounds may be generated in real time for the virtual character in the video based at least in part on modifying at least one of the plurality of parameters of each representation.

Voice generation for virtual characters

A system obtains scene context data characterizing the environment. The scene context data includes data that characterizes a trajectory of an agent in a vicinity of a vehicle up to a current time point. The system identifies a plurality of initial target locations, and generates, for each of a plurality of target locations that each corresponds to one of the initial target locations, a respective predicted likelihood score that represents a likelihood that the target location will be an intended final location for a future trajectory of the agent. For each target location in a first subset of the target locations, the system generates a predicted future trajectory for the agent given that the target location is the intended final location for the future trajectory. The system further selects, as likely future trajectories of the agent, one or more of the predicted future trajectories.

Agent trajectory prediction using target locations

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for agent trajectory prediction using vectorized inputs.

Agent trajectory prediction using vectorized inputs

The present disclosure describes techniques for facilitating interaction with content using interactive effects. The techniques comprise receiving a video from a first computing device associated with a first user among a plurality of users. The video is generated using an interactive effect. The interactive effect comprises overlaying at least one component on at least one subset of frames of the video. The interactive effect enables the video to be interacted with other users among the plurality of users. The video is presented on a second computing device associated with a second user among the plurality of users. The video comprises information indicative of instructions on how to interact with the video. Display of an animation associated with the at least one component may be automatically caused on the second computing device in response to receiving an indication of interacting with the video by the second user.

Content creation using interactive effects

A computer-implemented method for classifying video data with improved accuracy includes obtaining, by a computing system comprising one or more computing devices, video data comprising a plurality of video frames; extracting, by the computing system, a plurality of video tokens from the video data, the plurality of video tokens comprising a representation of spatiotemporal information in the video data; providing, by the computing system, the plurality of video tokens as input to a video understanding model, the video understanding model comprising a video transformer encoder model; and receiving, by the computing system, a classification output from the video understanding model.

Systems and methods for improved video understanding

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for determining summary content for resources in a domain. In one aspect, a method includes accessing a first resource belonging to a particular domain, selecting an anchor in the first resource linking to a second resource belonging to the particular domain, identifying particular text content in the first resource that is subordinate to the anchor that the second resource includes the particular text content that is subordinate to the anchor, based on determining that the second resource includes the particular text content that is subordinate to the anchor, generating a domain template for the particular domain, the domain template specifying a location of the particular text content in the second resource, and determining, for each respective resource belonging to the particular domain having a structure matching the domain template, respective text content for the respective resource.

Identifying information using referenced text

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for realizing a multimodal image classifier. In an aspect, a method includes, for each image of a plurality of images: processing the image by a textual generator model to obtain a set of phrases that are descriptive of the content of the image, wherein each phrase is one or more terms, processing the set of phrases by a textual embedding model to obtain an embedding of predicted text for the image, and processing the image using an image embedding model to obtain an embedding of image pixels of the image. Then a multimodal image classifier is trained on the embeddings of predicted text for the images and the embeddings of image pixels for the images to produce, as output, labels of an output taxonomy to classify an image based on the image as input.

Multimodal image classifier using textual and visual embeddings

Systems and methods for generating a virtual article of clothing at a display are described. Some examples may include: obtaining video data and audio data, analyzing the video data to determine one or more body joints of a target object appearing in the video data. A mesh based on the determined one or more body joints may be generated. The audio data may be analyzed to determine audio characteristics associated with the audio data. Texture rendering information associated with a virtual article of clothing may be determined based on the audio characteristics. A rendered video may be generated by rendering the virtual article of clothing to the generated mesh using the texture rendering information.

Rendering virtual articles of clothing based on audio characteristics

Systems and methods for rendering a video effect to a display are described. More specifically, video data and audio data are obtained. The video data is analyzed to determine one or more attachment points of a target object that appears in the video data. The audio data is analyzed to determine audio characteristics. A video effect associated with an animation to be added to the one or more attachment points is determined based on the audio characteristics. A rendered video is generated by applying the video effect to the video data.

Animation effect attachment based on audio characteristics

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for generating vehicle intent predictions using a neural network. One of the methods includes obtaining an input characterizing one or more vehicles in an environment; generating, from the input, features of each of the vehicles; and for each of the vehicles: processing the features of the vehicle using each of a plurality of intent-specific neural networks, wherein each of the intent-specific neural networks corresponds to a respective intent from a set of intents, and wherein each intent-specific neural network is configured to process the features of the vehicle to generate an output for the corresponding intent.

Vehicle intent prediction neural network

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for performing action localization in images and videos. In one aspect, a system comprises a data processing apparatus; a memory in data communication with the data processing apparatus and storing instructions that cause the data processing apparatus to perform image processing and video processing operations comprising: receiving an input comprising an image depicting a person; identifying a plurality of context positions from the image; determining respective feature representations of each of the context positions; providing a feature representation of the person and the feature representations of each of the context positions to a context neural network to obtain relational features, wherein the relational features represent relationships between the person and the context positions; and determining an action performed by the person using the feature representation of the person and the relational features.

Action localization in images and videos using relational features

Aspects of the disclosure relate to training and using a model for identifying actions of objects. For instance, LIDAR sensor data frames including an object bounding box corresponding to an object as well as an action label for the bounding box may be received. Each sensor frame is associated with a timestamp and is sequenced with respect to other sensor frames. Each given sensor data frame may be projected into a camera image of the object based on the timestamp associated with the given sensor data frame in order to provide fused data. The model may be trained using the fused data such that the model is configured to, in response to receiving fused data, the model outputs an action label for each object bounding box of the fused data. This output may then be used to control a vehicle in an autonomous driving mode.

Object action classification for autonomous vehicles

Growing community of inventors

San Francisco, CA, United States of America

Chen Sun