Google Inc.

Ankur P Parikh

Dipanjan Das

Thibault Sellam

Thomas Mieczyslaw Kwiatkowski

Swabha Swayamdipta

Systems and methods for automatic evaluation of the quality of NLG outputs. In some aspects of the technology, a learned evaluation model may be pretrained first using NLG model pretraining tasks, and then with further pretraining tasks using automatically generated synthetic sentence pairs. In some cases, following pretraining, the evaluation model may be further fine-tuned using a set of human-graded sentence pairs, so that it learns to approximate the grades allocated by the human evaluators.

Learned evaluation model for grading quality of natural language generation outputs

Systems and methods for automatic evaluation of the quality of NLG outputs. In some aspects of the technology, a learned evaluation model may be pretrained first using NLG model pretraining tasks, and then with further pretraining tasks using automatically generated synthetic sentence pairs. In some cases, following pretraining, the evaluation model may be further fine-tuned using a set of human-graded sentence pairs, so that it learns to approximate the grades allocated by the human evaluators. In some cases, following fine-tuning, the learned evaluation model may be distilled into a student model.

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for selecting a text span from an input electronic document that answers an input question. One of the methods includes obtaining a respective first numeric representations of text spans in the input document; for each of the text spans: determining, for a segment that contains the text span, a question-aware segment vector, determining, for the question, a segment-aware question vector, and processing the first numeric representation of the text span, the question-aware segment vector, and the segment-aware question vector using a second feedforward neural network to generate a second numeric representation of the text span; for each unique text span in the plurality of text spans: determining an aggregate representation for the unique text span, and determining, from the aggregate representation, a final score for the unique text span; and selecting a unique text span.

Growing community of inventors

New York, NY, United States of America

Ankur P Parikh