Google Inc.

Gdm Holding LLC

Mostafa Dehghani

Georg Heigold

Oriol Vinyals

Jakob David Uszkoreit

Lukasz Mieczyslaw Kaiser

Sylvain Gelly

Alexey Dosovitskiy

Jakob D Uszkoreit

Neil Matthew Tinmouth Houlsby

Dirk Weissenborn

Matthias Johannes Lorenz Minderer

Stephan Gouws

Lucas Klaus Beyer

Alexander Kolesnikov

Xiaohua Zhai

Thomas Unterthiner

Yang Li

Chen Sun

Xin Zhou

Emiel Hoogeboom

Cordelia Luise Schmid

Gang Li

Alexey Alexeevich Gritsenko

Mario Lucic

Dara Bahri

Donald Arthur Metzler, Jr

Vinh Quoc Tran

Anurag Arnab

Phillip Lippe

Jai Prakash Gupta

Jonathan Heek

Yi Tay

Adam Joshua Fisch

Tal Schuster

A computer-implemented method of generating multimodal data. The method comprises using a token generation neural network to generate, autoregressively, an output sequence of multimodal tokens, and in response to a next multimodal token being a start-of-image token, generating an image using an image generation subsystem conditioned on features representing the current sequence of multimodal tokens obtained from the token generation neural network. The method further comprises processing the image to convert pixels of the image into a sequence of image tokens, each image token comprising a block encoding of values of the pixels in a different region of the image that maps a set of values of the pixels to a respective image token, and appending the sequence of image tokens to the current output sequence of multimodal tokens as the next multimodal tokens in the output sequence of multimodal tokens.

Systems and methods for generating multimodal data using a single-tower architecture with a data generation subsystem

A method includes receiving, via a computing device, a screenshot of a display provided by a graphical user interface of the computing device. The method also includes generating, by an image-structure transformer of a neural network, a representation by fusing a first embedding based on the screenshot and a second embedding based on a layout of virtual objects in the screenshot. The method additionally includes predicting, by the neural network and based on the generated representation, a modeling task output associated with the graphical user interface. The method further includes providing, by the computing device, the predicted modeling task output.

Neural networks based multimodal transformer for multi-task user interface modeling

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for processing images using self-attention based neural networks. One of the methods includes obtaining one or more images comprising a plurality of pixels; determining, for each image of the one or more images, a plurality of image patches of the image, wherein each image patch comprises a different subset of the pixels of the image; processing, for each image of the one or more images, the corresponding plurality of image patches to generate an input sequence comprising a respective input element at each of a plurality of input positions, wherein a plurality of the input elements correspond to respective different image patches; and processing the input sequences using a neural network to generate a network output that characterizes the one or more images, wherein the neural network comprises one or more self-attention neural network layers.

Processing images using self-attention based neural networks

A computer-implemented method for classifying video data with improved accuracy includes obtaining, by a computing system comprising one or more computing devices, video data comprising a plurality of video frames; extracting, by the computing system, a plurality of video tokens from the video data, the plurality of video tokens comprising a representation of spatiotemporal information in the video data; providing, by the computing system, the plurality of video tokens as input to a video understanding model, the video understanding model comprising a video transformer encoder model; and receiving, by the computing system, a classification output from the video understanding model.

Systems and methods for improved video understanding

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for generating output sequences using auto-regressive decoder neural networks. In particular, during generation, adaptive early exiting is used to reduce the time required to generate the output sequence.

Efficient decoding of output sequences using adaptive early exiting

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for implementing a sequence to sequence model that is recurrent in depth while employing self-attention to combine information from different parts of sequences.

Universal transformers

Growing community of inventors

Amsterdam, Netherlands

Mostafa Dehghani