Google Inc.

Dipanjan Das

Ankur P Parikh

Thibault Sellam

Kuzman Ganchev

Jason Edward Weston

Fernando Pereira

Slav Petrov

Manaal Faruqui

Karl Moritz Hermann

Amarnag Subramanya

Ryan T McDonald

Keith Hall

Terry Yang-Hoe Koo

Oscar Carl Tackstrom

Joakim Nivre

Systems and methods for automatic evaluation of the quality of NLG outputs. In some aspects of the technology, a learned evaluation model may be pretrained first using NLG model pretraining tasks, and then with further pretraining tasks using automatically generated synthetic sentence pairs. In some cases, following pretraining, the evaluation model may be further fine-tuned using a set of human-graded sentence pairs, so that it learns to approximate the grades allocated by the human evaluators.

Learned evaluation model for grading quality of natural language generation outputs

Systems and methods for automatic evaluation of the quality of NLG outputs. In some aspects of the technology, a learned evaluation model may be pretrained first using NLG model pretraining tasks, and then with further pretraining tasks using automatically generated synthetic sentence pairs. In some cases, following pretraining, the evaluation model may be further fine-tuned using a set of human-graded sentence pairs, so that it learns to approximate the grades allocated by the human evaluators. In some cases, following fine-tuning, the learned evaluation model may be distilled into a student model.

Techniques are described herein for training and/or utilizing a query canonicalization system. In various implementations, a query canonicalization system can include a classification model and a canonicalization model. A classification model can be used to determine if a search query is well-formed. Additionally or alternatively, a canonicalization model can be used to determine a well-formed variant of a search query in response to determining a search query is not well-formed. In various implementations, a canonicalization model portion of a query canonicalization system can be a sequence to sequence model.

Canonicalizing search queries to natural language questions

A computer-implemented technique can include receiving, at a server, labeled training data including a plurality of groups of words, each group of words having a predicate word, each word having generic word embeddings. The technique can include extracting, at the server, the plurality of groups of words in a syntactic context of their predicate words. The technique can include concatenating, at the server, the generic word embeddings to create a high dimensional vector space representing features for each word. The technique can include obtaining, at the server, a model having a learned mapping from the high dimensional vector space to a low dimensional vector space and learned embeddings for each possible semantic frame in the low dimensional vector space. The technique can also include outputting, by the server, the model for storage, the model being configured to identify a specific semantic frame for an input.

Semantic frame identification with distributed word representations

A computer-implemented method can include obtaining (i) an aligned bi-text for a source language and a target language, and (ii) a supervised sequence model for the source language. The method can include labeling a source side of the aligned bi-text using the supervised sequence model and projecting labels from the labeled source side to a target side of the aligned bi-text to obtain a labeled target side of the aligned bi-text. The method can include filtering the labeled target side based on a task of a natural language processing (NLP) system configured to utilize a sequence model for the target language to obtain a filtered target side of the aligned bi-text. The method can also include training the sequence model for the target language using posterior regularization with soft constraints on the filtered target side to obtain a trained sequence model for the target language.

Cross-lingual discriminative learning of sequence models with posterior regularization

Systems and techniques are provided for training a natural language processing model with information retrieval model annotations. A natural language processing model may be trained, through machine learning, using training examples that include part-of-speech tagging and annotations added by an information retrieval model. The natural language processing model may generate part-of-speech, parse-tree, beginning, inside, and outside label, mention chunking, and named-entity recognition predictions with confidence scores for text in the training examples. The information retrieval model annotations and part-of-speech tagging in the training example may be used to determine the accuracy of the predictions, and the natural language processing model may be adjusted. After training, the natural language processing model may be used to make predictions for novel input, such as search queries and potential search results. The search queries and potential search results may have information retrieval model annotations.

Training a natural language processing model with information retrieval model annotations

A method and system are provided for a part-of-speech tagger that may be particularly useful for resource-poor languages. Use of manually constructed tag dictionaries from dictionaries via bitext can be used as type constraints to overcome the scarcity of annotated data in some instances. Additional token constraints can be projected from a resource-rich source language via word-aligned bitext. Several example models are provided to demonstrate this such as a partially observed conditional random field model, where coupled token and type constraints may provide a partial signal for training. The disclosed method achieves a significant relative error reduction over the prior state of the art.

Weakly supervised part-of-speech tagging with coupled token and type constraints

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for graph-based semi-supervised learning of structured tagging models. In one aspect, a method includes creating a graph having a plurality of unique vertices in which vertices in a first set of vertices represent n-grams that are each associated with a respective part-of-speech and that were derived from labeled source domain text, and in which vertices in a different second set of vertices represent n-grams that are not associated with a part-of-speech and that were derived from unlabeled target domain text. A respective measure of similarity is calculated between the vertices in each of the pairs based at least partially on a distance between the respective features of the pair used to weight a graph edge between the pair.

Growing community of inventors

Jersey City, NJ, United States of America

Dipanjan Das