Systems for encoding documents and generating semantic identifiers

Abstract

A semantic identifier that represents information within a document may be determined by training a language model or other machine learning model using a self-supervised process. The model encodes text to generate an embedding. A decoder determines a semantic identifier token that represents information in the text based on the embedding and a previous semantic identifier token if present. The identifier determined by the decoder is aligned with one of a selected set of semantic identifiers indicated in codebook data. To determine the accuracy of the semantic identifier token, a second decoder is used to attempt to reconstruct the original text using the semantic identifier token and a portion of the original text. Differences between the reconstructed text and the original text are used to determine a loss value, and the parameters of the machine learning model or the codebook data are trained based on minimizing the loss value.