Google Inc.

Intel Corporation

Jasper Display Corporation

Oracle International Corporation

Peoplesoft, Inc.

Bo Li

Tara N Sainath

Ron J Weiss

Kaushik Sheth

Yonghui Wu

Michiel A U Bacchiani

Shuo-Yiin Chang

Ruoming Pang

Zhifeng Chen

Edwin Lyle Hudson

Patrick An Phu Nguyen

Rohit Prakash Prabhavalkar

Kevin William Wilson

Yanzhang He

Khe Chai Sim

Pedro Jose Moreno Mengibar

Raymond W Zeng

Chung-Cheng Chiu

Eugene Weinstein

Gabor Simko

Shuo-yiin Chang

Yu Zhang

Mase J Taub

Navdeep Jaitly

Trevor D Strohman

Arun Narayanan

Maria Carolina Parada San Martin

Kanury Kanishka Rao

David Charles McDonald

Sean Matthew Shannon

Wei Han

Qiao Liang

Anjuli Patricia Kannan

Yang Li

Zelin Wu

Chun Xia

Chao Zhang

Yufeng Li

Johnny Hsiang Wu

Ekaterina Gonina

James Michael Stover

Jeffrey Tang Fung Li

Victor S Zhu

Jan Kazimierz Chorowski

Deepti Bhatia

Chandra V Mouli

Wei Ping Li

Rajesh Sundaram

Kerry Dean Tedrow

Sandeep K Guliani

Antoine Jean Bruguier

Wei Li

Matthew Goldman

Rezaul Haque

Ankur Bapna

Parisa Haghani

David Rybach

Jiahui Yu

Ke Hu

Saad Monasa

Marc E Landgraf

Guru Prakash Arumugam

Anmol Gulati

Priya Walimbe

Liangliang Cao

Yangzhang He

Manaal Faruqui

Qiumin Xu

Edward M Babb

Ian Mcgraw

Zhiyun Lu

Ahsanur Rahman

David Qiu

Qiujia Li

Aishwarya Varadhan

Murali Krishna

Ding Zhao

Dung C Nguyen

Sameer Bidichandani

Heguang Liu

Shyam Upadhyay

Vince Chen

Nikhil Siddhartha

Junwen Bai

Jahanshier Javanifard

Adam Stambler

Ross A Giddings

Subramanyam Chandramouli

Ahmed Muneer

Michiel AU Bacchiani

Systems and methods of the present disclosure are directed to a computing system, including one or more processors and a machine-learned multi-mode speech recognition model configured to operate in a streaming recognition mode or a contextual recognition mode. The computing system can perform operations including obtaining speech data and a ground truth label and processing the speech data using the contextual recognition mode to obtain contextual prediction data. The operations can include evaluating a difference between the contextual prediction data and the ground truth label and processing the speech data using the streaming recognition mode to obtain streaming prediction data. The operations can include evaluating a difference between the streaming prediction data and the ground truth label and the contextual and streaming prediction data. The operations can include adjusting parameters of the speech recognition model.

Systems and methods for training dual-mode machine-learned speech recognition models

A memory cell for a display is disclosed. The memory cell has a current limiter on the power supply to reduce the power consumed by the memory cell during a write operation when the binary state of the memory cell is flipped. In a dense memory environment, in a display with a million or more memory cells, the incremental power reduction of each memory cell corresponds to a substantial reduction in the overall power consumed by the display.

Memory cell for a pixel of a display

A method includes receiving a sequence of acoustic frames characterizing one or more utterances. At each of a plurality of output steps, the method also includes generating, by an encoder network of a speech recognition model, a higher order feature representation for a corresponding acoustic frame of the sequence of acoustic frames, generating, by a prediction network of the speech recognition model, a hidden representation for a corresponding sequence of non-blank symbols output by a final softmax layer of the speech recognition model, and generating, by a first joint network of the speech recognition model that receives the higher order feature representation generated by the encoder network and the dense representation generated by the prediction network, a probability distribution that the corresponding time step corresponds to a pause and an end of speech.

Disfluency detection models for natural conversational voice systems

A method includes receiving, as input to a speech recognition model, audio data corresponding to a spoken utterance. The method also includes performing, using the speech recognition model, speech recognition on the audio data by, at each of a plurality of time steps, encoding, using an audio encoder, the audio data corresponding to the spoken utterance into a corresponding audio encoding, and decoding, using a speech recognition joint network, the corresponding audio encoding into a probability distribution over possible output labels. At each of the plurality of time steps, the method also includes determining, using an intended query (IQ) joint network configured to receive a label history representation associated with a sequence of non-blank symbols output by a final softmax layer, an intended query decision indicating whether or not the spoken utterance includes a query intended for a digital assistant.

Intended query detection using E2E modeling for continued conversation

Methods, systems, and apparatus, including computer programs encoded on a computer-readable media, for speech recognition using multi-dialect and multilingual models. In some implementations, audio data indicating audio characteristics of an utterance is received. Input features determined based on the audio data are provided to a speech recognition model that has been trained to output score indicating the likelihood of linguistic units for each of multiple different language or dialects. The speech recognition model can be one that has been trained using cluster adaptive training. Output that the speech recognition model generated in response to receiving the input features determined based on the audio data is received. A transcription of the utterance generated based on the output of the speech recognition model is provided.

Multi-dialect and multilingual speech recognition

A method includes receiving audio features and generating a latent speech representation based on the audio features. The method also includes generating a target quantized vector token and a target token index for a corresponding latent speech representation. The method also includes generating a contrastive context vector for a corresponding unmasked or masked latent speech representation and deriving a contrastive self-supervised loss based on the corresponding contrastive context vector and the corresponding target quantized vector token. The method also include generating a high-level context vector based on the contrastive context vector and, for each high-level context vector, learning to predict the target token index at the corresponding time step using a cross-entropy loss based on the target token index. The method also includes predicting speech recognition hypotheses for the utterance and training a multilingual automatic speech recognition (ASR) model using an unsupervised loss and a supervised loss.

Joint unsupervised and supervised training for multilingual ASR

A display system comprising a plurality of display controller circuits controlling a like number of independent segments of pixel drive circuits of a backplane. Each pixel drive circuit comprises a memory element and associated pixel drive circuitry. The segments of the backplane may be organized vertically. The word line for the memory cells of a first segment of pixel drive circuits passes underneath a second segment of pixel drive circuits without directly interacting with the pixel drive circuits of the second segment in order to reach the pixel drive circuits of the first segment. The plurality of display controller circuits operate asynchronously but are kept at the same frame rate by an external signal such as Vsync.

Larger backplane suitable for high speed applications

A speech recognition model includes an encoder network, a prediction network, and a joint network. The encoder network is configured to receive a sequence of acoustic frames characterizing an input utterance; and generate, at each of a plurality of output steps, a higher order feature representation for a corresponding acoustic frame in the sequence of acoustic frames. The prediction network is configured to: receive a sequence of non-blank symbols output by a final Softmax layer; and generate, at each of the plurality of output steps, a dense representation. The joint network is configured to generate, at each of the plurality of output steps based on the higher order feature representation and the dense representation, a probability distribution over possible speech recognition hypotheses. The joint network includes a stack of gating and bilinear pooling to fuse the dense representation and the higher order feature representation.

Fusion of acoustic and text representations in RNN-T

A method includes receiving a sequence of acoustic frames characterizing one or more utterances as input to a multilingual automated speech recognition (ASR) model. The method also includes generating a higher order feature representation for a corresponding acoustic frame. The method also includes generating a hidden representation based on a sequence of non-blank symbols output by a final softmax layer. The method also includes generating a probability distribution over possible speech recognition hypotheses based on the hidden representation generated by the prediction network at each of the plurality of output steps and the higher order feature representation generated by the encoder at each of the plurality of output steps. The method also includes predicting an end of utterance (EOU) token at an end of each utterance. The method also includes classifying each acoustic frame as either speech, initial silence, intermediate silence, or final silence.

Language agnostic multilingual end-to-end streaming on-device ASR system

A method for performing speech recognition using sequence-to-sequence models includes receiving audio data for an utterance and providing features indicative of acoustic characteristics of the utterance as input to an encoder. The method also includes processing an output of the encoder using an attender to generate a context vector, generating speech recognition scores using the context vector and a decoder trained using a training process, and generating a transcription for the utterance using word elements selected based on the speech recognition scores. The transcription is provided as an output of the ASR system.

Speech recognition with sequence-to-sequence models

A computer-implemented method of training a streaming speech recognition model that includes receiving, as input to the streaming speech recognition model, a sequence of acoustic frames. The streaming speech recognition model is configured to learn an alignment probability between the sequence of acoustic frames and an output sequence of vocabulary tokens. The vocabulary tokens include a plurality of label tokens and a blank token. At each output step, the method includes determining a first probability of emitting one of the label tokens and determining a second probability of emitting the blank token. The method also includes generating the alignment probability at a sequence level based on the first probability and the second probability. The method also includes applying a tuning parameter to the alignment probability at the sequence level to maximize the first probability of emitting one of the label tokens.

Fast emit low-latency streaming ASR with sequence-level emission regularization utilizing forward and backward probabilities between nodes of an alignment lattice

Two-pass automatic speech recognition (ASR) models can be used to perform streaming on-device ASR to generate a text representation of an utterance captured in audio data. Various implementations include a first-pass portion of the ASR model used to generate streaming candidate recognition(s) of an utterance captured in audio data. For example, the first-pass portion can include a recurrent neural network transformer (RNN-T) decoder. Various implementations include a second-pass portion of the ASR model used to revise the streaming candidate recognition(s) of the utterance and generate a text representation of the utterance. For example, the second-pass portion can include a listen attend spell (LAS) decoder. Various implementations include a shared encoder shared between the RNN-T decoder and the LAS decoder.

Two-pass end to end speech recognition

A plurality of pixel drive circuits form part of an array of emissive elements. The plurality of pixel drive circuits are disposed to form a plurality of rows and a plurality of columns. The plurality of pixel drive circuits are organized into sets of pixel drive circuits, and each set comprises at least one pixel drive circuit. A FET of a set of pixel drive circuits shares a common well with other FETs of similar function in the same set of pixel drive circuits positioned therein, such that the variance of the threshold voltages of those FETs is substantially reduced. Each of the pixel drive circuits comprises a circuit operative to deliver a current at a predetermined voltage to an emissive device and a memory circuit operative to receive modulation data and to use same to modulate the current output of the pixel drive circuit.

Backplane for an array of emissive elements

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for neural network adaptive beamforming for multichannel speech recognition are disclosed. In one aspect, a method includes the actions of receiving a first channel of audio data corresponding to an utterance and a second channel of audio data corresponding to the utterance. The actions further include generating a first set of filter parameters for a first filter based on the first channel of audio data and the second channel of audio data and a second set of filter parameters for a second filter based on the first channel of audio data and the second channel of audio data. The actions further include generating a single combined channel of audio data. The actions further include inputting the audio data to a neural network. The actions further include providing a transcription for the utterance.

Adaptive audio enhancement for multichannel speech recognition

Growing community of inventors

Santa Clara, CA, United States of America

Bo Li