Retrieval-augmented metadata generation

Abstract

Here is schema generation from a data table and reference documentation, including inferential analysis and generation of natural language. A large language model (LLM) inferentially generates an initial column summary that consists of natural language that describes exactly one column of a data table. From a vector store that contains many fixed-size portions of many reference documents, a few nearest neighbor portions are dense retrieved that are semantically similar to the initial column summary. From the initial column summary and the nearest neighbor portions, the LLM inferentially generates an expanded column summary that describes the column, including inferring that some nearest neighbor portions should be excluded. From the expanded column summary, the LLM inferentially generates a) a data schema that consists of semi-structured text that describes the data table, including a column comment that contains natural language that describes the column.