Llm-based recommender system for data catalog

Abstract

LLMs use probabilistic methods to create coherent responses, sometimes going beyond the training data. This can result in 'LLM hallucination.' Asset metadata is used in constructing prompts for the LLM, enhancing the LLM's understanding of available assets. Including a limited set of candidate assets in an LLM prompt template as part of a chain-of-thought prompt can solve the hallucination issue. Multiple rounds of interaction with the LLM may be used, allowing for more dynamic and responsive user engagement. The LLM-based recommender system for data catalogs may comprise asset metadata, user data, and a recommender model. The output of the LLM-based recommender system is a recommended asset list. The prompts may explicitly instruct the LLM to limit elements of the list to the candidate set of assets. The LLM's advanced context understanding and reasoning abilities enable it to deliver accurate and interpretable personalized recommendations.