Systems and methods for training language models to reason over tables

Abstract

Systems and methods for pre-training and fine-tuning of neural-network-based language models to reason directly over tables without generating logical forms. In some examples, a language model can be pre-trained using masked-language modeling tasks synthetically generated from tables pulled from a knowledge corpus. In some examples, the language model may be further pre-trained using pairs of counterfactual statements generated from those tables, and/or one or more statements that compare selected data from those tables. The language model may then be fine-tuned using examples that include only a question, an answer, and a table, allowing fine-tuning examples to be harvested directly from existing benchmark datasets or synthetically generated.