Enhanced object detection with retrieval augmented generation and language model prompting system

Abstract

Certain aspects of the disclosure provide a method for enhanced object detection. The method includes providing, to a machine learning (ML) model, a first prompt comprising a first image and a first instruction to output a first description; obtaining, from the ML model, the first description comprising an identification of an unidentified visual object; generating an embedding of the unidentified visual object; obtaining, from a retrieval augmented generation (RAG) database, an embedding associated with a known visual object and satisfying a similarity threshold; retrieving information associated with the known visual object; generating an enhanced context comprising the information associated with the known visual object; providing, to the ML model, a second prompt comprising the enhanced context and a second instruction to output a second description of the first image; and obtaining, from the ML model, the second description including an identification of a visual object associated with the unidentified visual object.