Multimodal retrieval augmented generation

Abstract

A retrieval augmented generation (RAG) system determines, based on a query, an audio embedding type or a text embeddings type to use to filter an embeddings store, wherein the embedding store includes embeddings of the audio embedding type and the text embeddings type in a vector space. The RAG system obtains, from a filtered portion of the embeddings store, embeddings based on similarity scores between the plurality of embeddings and a representation of the query in the vector space. The RAG system obtains embedding source data from which a selected one of the plurality of embeddings of the embeddings store have been generated, wherein the embedding source data is source audio data or transcribed source audio data used to generate the embeddings in the embeddings store. The RAG system provides the query and the embedding source data for the selected embedding to the multimodal LLM.