Google Inc.

Kevin Stefan Clark

Quoc V Le

Thang Minh Luong

Systems and methods are provided that train a machine-learned language encoding model through the use of a contrastive learning task. In particular, the present disclosure describes a contrastive learning task where the encoder learns to distinguish input tokens from plausible alternatives. In some implementations, on each training example the proposed method masks out some subset (e.g., 15%) of the original input tokens, replaces the masked tokens with samples from a 'generator' (e.g., which may be a small masked language model), and then trains the encoder to predict whether each token comes from the original data or is a replacement produced by the generator.

Contrastive pre-training for language tasks

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for training a machine learning model using teacher annealing.

Growing community of inventors

Redwood City, CA, United States of America

Kevin Stefan Clark