Salesforce Com, Inc.

Romain Paulus

Caiming Xiong

Richard Socher

Wojciech Kryscinski

Stephen Merity

Ankit Kumar

Ozan Irsoy

Mohit Iyyer

A novel unified neural network framework, the dynamic memory network, is disclosed. This unified framework reduces every task in natural language processing to a question answering problem over an input sequence. Inputs and questions are used to create and connect deep memory sequences. Answers are then generated based on dynamically retrieved memories.

Dynamic memory network

A system for text summarization includes an encoder for encoding input tokens of a document and a decoder for emitting summary tokens which summarize the document based on the encoded input tokens. At each iteration the decoder generates attention scores between a current hidden state of the decoder and previous hidden states of the decoder, generates a current decoder context from the attention scores and the previous hidden states of the decoder, and selects a next summary token based on the current decoder context. The selection of the next summary token prevents emission of repeated summary phrases in a summary of the document.

Deep reinforced model for abstractive summarization

A system is disclosed for providing an abstractive summary of a source textual document. The system includes an encoder, a decoder, and a fusion layer. The encoder is capable of generating an encoding for the source textual document. The decoder is separated into a contextual model and a language model. The contextual model is capable of extracting words from the source textual document using the encoding. The language model is capable of generating vectors paraphrasing the source textual document based on pre-training with a training dataset. The fusion layer is capable of generating the abstractive summary of the source textual document from the extracted words and the generated vectors for paraphrasing. In some embodiments, the system utilizes a novelty metric to encourage the generation of novel phrases for inclusion in the abstractive summary.

Abstraction of text summarization

A system for text summarization includes an encoder for encoding input tokens of a document and a decoder for emitting summary tokens which summarize the document based on the encoded input tokens. At each iteration the decoder generates attention scores between a current hidden state of the decoder and previous hidden states of the decoder, generates a current decoder context from the attention scores and the previous hidden states of the decoder, and selects a next summary token based on the current decoder context and a current encoder context of the encoder. The attention scores penalize candidate summary tokens having high attention scores in previous iterations. In some embodiments, the attention scores include an attention score for each of the previous hidden states of the decoder. In some embodiments, the selection of the next summary token prevents emission of repeated summary phrases in a summary of the document.

Disclosed RNN-implemented methods and systems for abstractive text summarization process input token embeddings of a document through an encoder that produces encoder hidden states; applies the decoder hidden state to encoder hidden states to produce encoder attention scores for encoder hidden states; generates encoder temporal scores for the encoder hidden states by exponentially normalizing a particular encoder hidden state's encoder attention score over its previous encoder attention scores; generates normalized encoder temporal scores by unity normalizing the temporal scores; produces the intra-temporal encoder attention vector; applies the decoder hidden state to each of previous decoder hidden states to produce decoder attention scores for each of the previous decoder hidden states; generates normalized decoder attention scores for previous decoder hidden states by exponentially normalizing each of the decoder attention scores; identifies previously predicted output tokens; produces the intra-decoder attention vector and processes the vector to emit a summary token.

Growing community of inventors

Menlo Park, CA, United States of America

Romain Paulus