Google Inc.

Alexey Alexeevich Gritsenko

Alexey Dosovitskiy

Neil Matthew Tinmouth Houlsby

Dirk Weissenborn

Austin Charles Stone

Matthias Johannes Lorenz Minderer

Yang Li

Xin Zhou

Mostafa Dehghani

Tim Salimans

Gang Li

A method includes receiving, via a computing device, a screenshot of a display provided by a graphical user interface of the computing device. The method also includes generating, by an image-structure transformer of a neural network, a representation by fusing a first embedding based on the screenshot and a second embedding based on a layout of virtual objects in the screenshot. The method additionally includes predicting, by the neural network and based on the generated representation, a modeling task output associated with the graphical user interface. The method further includes providing, by the computing device, the predicted modeling task output.

Neural networks based multimodal transformer for multi-task user interface modeling

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for object detection. In one aspect, a method comprises: obtaining: (i) an image, and (ii) a set of one or more query embeddings, wherein each query embedding represents a respective category of object; processing the image and the set of query embeddings using an object detection neural network to generate object detection data for the image, comprising: processing the image using an image encoding subnetwork of the object detection neural network to generate a set of object embeddings; processing each object embedding using a localization subnetwork to generate localization data defining a corresponding region of the image; and processing: (i) the set of object embeddings, and (ii) the set of query embeddings, using a classification subnetwork to generate, for each object embedding, a respective classification score distribution over the set of query embeddings.

Open-vocabulary object detection in images

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for training a generative neural network to convert conditioning text inputs to audio outputs using energy scores.

Training speech synthesis neural networks using energy scores

Growing community of inventors

Amsterdam, Netherlands

Alexey Alexeevich Gritsenko