Detection and classification based on scene embeddings and image embeddings

Abstract

Techniques for detecting and/or classifying objects in a driving environment are described herein. A vehicle may receive sensor data and input the sensor data into a perception and/or localization component which may be trained to provide processed output data (e.g., bounding boxes, track(s), etc.). In such cases, the vehicle may generate a scene embedding based on the output data. Further, the vehicle may receive image data from an image capturing device on the vehicle. The vehicle may generate an image embedding associated with the image data and concatenate the image embedding with the scene embedding. The vehicle may input the concatenated embedding into a detection and/or classification model to detect and/or classify object(s) within the environment.