Automated metadata generation and natural language database querying

Abstract

A machine-readable medium stores instructions that, when executed by a processor, cause the processor to execute operations. The operations include receiving information identifying a database table and preprocessing the table. The preprocessing includes extracting primary and foreign key relationships and generating sample data by combining frequency-based sampling of common values with random sampling. The operations also include generating, using an LLM, a unified metadata schema. The generating includes processing the table and the sample data using domain-specific prompts and outputting the unified metadata schema. The operations include embedding and indexing the unified metadata schema and converting a natural language question output from an end-user device into a SQL query. The converting includes identifying relevant metadata in the unified metadata schema matching query intent and generating the SQL query using the filtered metadata. The operations also include executing the SQL query and providing query results to the end-user device.