Vision-language-planning (vlp) models with agent-wise learning for autonomous driving

Abstract

Methods and systems for training an autonomous driving, agent-centric vison-language planning (VLP) machine learning model. Image data is obtained from a vehicle-mounted camera, encompassing details about agents situated within the external environment. Via image processing, the system identifies these agents within the environment. A Bird's Eye View (BEV) representation of the surroundings is then generated, encapsulating BEV features including spatiotemporal information linked to the vehicle and the recognized agents. Executing the VLP model begins by first extracting agent-wise BEV features from the BEV, wherein the agent-wise BEV features are associated with respective agents in the environment. Agent-wise text features are extracted from natural language text prompts. A contrastive learning model derives similarities between the agent-wise BEV features and the agent-wise text features. The BEV features are refined based on the similarities, and the refined BEV features are used in various models to generate a modified predicted trajectory of the vehicle.