Contrastive training of object trajectory encoders and text encoders

Abstract

Techniques are described herein for training contrastive models including object trajectory encoders and text encoders for evaluating, classifying, and/or predicting the movements and behaviors of dynamic objects in driving environments. A training system may receive sets of ground truth trajectory data describing movements of objects within driving environments, and associated text descriptions related to the trajectory data. The training system may jointly train the trajectory encoder and the text encoder, using contrastive loss, based on the related sets of trajectory data and text data. Once trained, the trajectory encoder and/or the text encoder may operate as pre-trained models for subsequently training and executing additional models with different output heads and/or various other downstream encoding tasks. In some examples, contrastive pre-trained trajectory encoders trained as described herein may be used for training and executing motion forecasting models within autonomous vehicles.