Google Inc.

Dmitry Lepikhin

Thang Minh Luong

Ankur Bapna

Orhan Firat

Yanping Huang

Maxim Krikun

Sneha Kudugunta

Systems and methods for routing in mixture-of-expert models. In some aspects of the technology, a transformer may have at least one Mixture-of-Experts ('MoE') layer in each of its encoder and decoder, with the at least one MoE layer of the encoder having a learned gating function configured to route each token of a task to two or more selected expert feed-forward networks, and the at least one MoE layer of the decoder having a learned gating function configured to route each task to two or more selected expert feed-forward networks.

Growing community of inventors

Sunnyvale, CA, United States of America

Dmitry Lepikhin