Google Inc.

Georg Heigold

Mostafa Dehghani

Alexey Dosovitskiy

Sylvain Gelly

Jakob D Uszkoreit

Neil Matthew Tinmouth Houlsby

Dirk Weissenborn

Xiaohua Zhai

Lucas Klaus Beyer

Alexander Kolesnikov

Matthias Johannes Lorenz Minderer

Thomas Unterthiner

Chen Sun

Cordelia Luise Schmid

Sara Sabour Rouh Aghdam

Austin Charles Stone

Gamaleldin Elsayed

Anurag Arnab

Mario Lucic

Thomas Kipf

Aravindh Mahendran

Klaus Greff

Rico Jonschkowski

A method includes obtaining first feature vectors and second feature vectors representing contents of a first and second image frame, respectively, of an input video. The method may also include generating, based on the first feature vectors, first slot vectors, where each slot vector represents attributes of a corresponding entity as represented in the first image frame, and generating, based on the first slot vectors, predicted slot vectors including a corresponding predicted slot vector that represents a transition of the attributes of the corresponding entity from the first to the second image frame. The method may additionally include generating, based on the predicted slot vectors and the second feature vectors, second slot vectors including a corresponding slot vector that represents the attributes of the corresponding entity as represented in the second image frame, and determining an output based on the predicted slot vectors or the second slot vectors.

Conditional object-centric learning with slot attention for video and other sequential data

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for processing images using self-attention based neural networks. One of the methods includes obtaining one or more images comprising a plurality of pixels; determining, for each image of the one or more images, a plurality of image patches of the image, wherein each image patch comprises a different subset of the pixels of the image; processing, for each image of the one or more images, the corresponding plurality of image patches to generate an input sequence comprising a respective input element at each of a plurality of input positions, wherein a plurality of the input elements correspond to respective different image patches; and processing the input sequences using a neural network to generate a network output that characterizes the one or more images, wherein the neural network comprises one or more self-attention neural network layers.

Processing images using self-attention based neural networks

A computer-implemented method for classifying video data with improved accuracy includes obtaining, by a computing system comprising one or more computing devices, video data comprising a plurality of video frames; extracting, by the computing system, a plurality of video tokens from the video data, the plurality of video tokens comprising a representation of spatiotemporal information in the video data; providing, by the computing system, the plurality of video tokens as input to a video understanding model, the video understanding model comprising a video transformer encoder model; and receiving, by the computing system, a classification output from the video understanding model.

Systems and methods for improved video understanding

Growing community of inventors

Aachen, Germany

Georg Heigold