Google Inc.

Peter Junteng Liu

Mohammad Saleh

Noam M Shazeer

Lukasz Mieczyslaw Kaiser

Ben David Goodrich

Ryan Sepassi

Etienne Pot

Andrew M Dai

Jingqing Zhang

Yao Zhao

Jeffrey Adgate Dean

Kai Chen

Quoc V Le

Yi Zhang

Kun Zhang

James Wilson

Gregory Sean Corrado

Yingwei Cui

Nissan Hajaj

Alvin Rishi Rajkomar

Benjamin Irvine

Eyal Oren

James Wexler

Patrik Sundberg

Hector Yee

Michaela Hardt

Xiaomi Sun

Xiaobing Liu

Jacob Marcus

Alexander Mossin

Ron J Weiss

Balint Miklos

Douglas Eck

Thang Minh Luong

Itamar Gilad

Karol Piotr Kurach

Colin Abraham Raffel

Ivo Krka

Kurt Litsch

Gerardo Flores

Gavin Duggan

Liam MacDermed

Alexandru Damian

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for generating an output sequence from an input sequence. One of the methods includes, at each of a plurality of generation time steps: generating a combined sequence for the generation time step that includes the input sequence followed by the output tokens that have already been generated as of the generation time step; processing the combined sequence using a self-attention decoder neural network to generate a time step output that defines a score distribution over a set of possible output tokens; and selecting, using the time step output, an output token from the set of possible output tokens as the next output token in the output sequence.

Attention-based decoder-only sequence transduction neural networks

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for training a text summarization neural network. One of the methods includes pre-training the text summarization neural network including learning values of a plurality of network parameters through self-supervised learning using unlabeled data comprising unlabeled first texts, the pre-training including: obtaining an unlabeled first text comprising a plurality of segments; selecting one or more of the plurality of segments; processing a masked first text that excludes the one or more selected segments to generate a prediction of the one or more selected segments; and determining, based on a difference between the prediction and the one or more selected segments, an update to the current values of the plurality of network parameters; adapting the pre-trained text summarization neural network for a specific text summarization task using labeled data comprising second texts and respective summaries of the second texts.

Training text summarization neural networks with an extracted segments prediction objective

A system for predicting and summarizing medical events from electronic health records includes a computer memory storing aggregated electronic health records from a multitude of patients of diverse age, health conditions, and demographics including medications, laboratory values, diagnoses, vital signs, and medical notes. The aggregated electronic health records are converted into a single standardized data structure format and ordered arrangement per patient, e.g., into a chronological order. A computer (or computer system) executes one or more deep learning models trained on the aggregated health records to predict one or more future clinical events and summarize pertinent past medical events related to the predicted events on an input electronic health record of a patient having the standardized data structure format and ordered into a chronological order. An electronic device configured with a healthcare provider-facing interface displays the predicted one or more future clinical events and the pertinent past medical events of the patient.

System and method for predicting and summarizing medical events from electronic health records

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for generating a target sequence including a respective output at each of multiple output time steps from respective encoded representations of inputs in an input sequence. The method includes, for each output time step, starting from the position, in the input order, of the encoded representation that was selected as a preceding context vector at a preceding output time step, traversing the encoded representations until an encoded representation is selected as a current context vector at the output time step. A decoder neural network processes the current context vector and a preceding output at the preceding output time step to generate a respective output score for each possible output and to update the hidden state of the decoder recurrent neural network. An output is selected for the output time step using the output scores.

Sequence processing using online attention

Systems and methods are provided for organizing messages. A plurality of messages is communicated to a user with a designation of the message category of each respective message in the plurality of messages. The plurality of messages includes, for each respective message category in a plurality of message categories, at least one message in the respective message category. A user selection of a first message in the plurality of messages is received, whereupon a subset of the plurality of messages similar to the first message is selected based upon respective comparisons of continuous vector representations of (i) a set of words in the first message and (ii) a corresponding set of words in messages in the subset of messages. An identification of each respective message in the subset of messages is then displayed.

Systems and methods for estimating message similarity

Growing community of inventors

Mountain View, CA, United States of America

Peter Junteng Liu