Next ast branch prediction and next token prediction joint pre-training task for code generative models

Abstract

During pretraining, a computer generates three trainable and untrained machine learning models that are a token sequence encoder, a token predictor, and a path predictor. A sequence of lexical tokens is generated that represents a lexical text in a training corpus. A graph is generated that represents the lexical text. In the graph, a next traversal path is selected that corresponds to a next lexical token that is adjacent to a sliding subsequence of the sequence of lexical tokens. From the subsequence, the token sequence encoder infers an encoded sequence that represents the subsequence. The path predictor and token predictor accept the encoded sequence as input for respective inferencing for which respective training losses are measured. Both training losses are combined into a combined loss that is used to increase the accuracy of the three machine learning models by, for example, backpropagation of the combined loss.