Google Inc.

Dara Bahri

Mostafa Dehghani

Donald Arthur Metzler, Jr

Yi Tay

Liu Yang

Da-Cheng Juan

Vinh Quoc Tran

Jai Prakash Gupta

Adam Joshua Fisch

Tal Schuster

The present disclosure provides echo-attention layers, a new efficient method for increasing the expressiveness of self-attention layers without incurring significant parameter or training time costs. One intuition behind the proposed method is to learn to echo, i.e., attend once and then get N echo-ed attentions for free (or at a relatively cheap cost). As compared to stacking new layers, the proposed echoed attentions are targeted at providing similar representation power at a better cost efficiency.

Machine-learned attention models featuring echo-attention layers

A system for performing a machine learning task on a network input is described. The system includes one or more computers and one or more storage devices storing instructions that, when executed by the one or more computers, cause the one or more computers to implement (i) multiple sorting networks in which each sorting network is configured to sort vector blocks in a sequence of vector blocks to generate a sorted sequence of vector blocks; and (ii) a sorting attention neural network configured to perform the machine learning task on the input sequence by executing multiple sorting attention mechanisms using the sorting networks.

Sorting attention neural networks

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for generating output sequences using auto-regressive decoder neural networks. In particular, during generation, adaptive early exiting is used to reduce the time required to generate the output sequence.

Efficient decoding of output sequences using adaptive early exiting

Growing community of inventors

Lafayette, CA, United States of America

Dara Bahri