Google Inc.

Joshua Timothy Ainslie

Manzil Zaheer

Amr Ahmed

Guru Guruganesh

Santiago Ontañón

Philip Pham

Kumar Avinava Dubey

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for processing network inputs using an attention neural network that has one or more sparse attention sub-layers. Each sparse attention sub-layer is configured to apply a sparse attention mechanism that attends differently for input positions that are in a first proper subset of the input positions in the input to the sub-layer than for positions that are not in the first proper subset.

Growing community of inventors

Saratoga, CA, United States of America

Joshua Timothy Ainslie