International Business Machines Corporation

Nuance Communications, Inc.

Toyota Technological Institute at Chicago

Oipenn, Inc.

Michael Alan Picheny

David Nahamoo

Lalit R Bahl

Ponani S Gopalakrishnan

Peter V De Souza

Bhuvana Ramabhadran

Samuel Jonathan Thomas

Yuqing Gao

George Andrei Saon

Robert L Mercer

Peter V DeSouza

Dimitri Kanevsky

Raimo Bakis

Kartik Audhkhasi

Mark Edward Epstein

Chengjun Julian Chen

Jerome R Bellegarda

Wael Mohamed Hamza

Miroslav Novak

Tara N Sainath

Sara H Basson

Andrew Jason Lavery

Kenney Ng

Hong-Kwang Jeff Kuo

Ellen Marie Eide

Brian E D Kingsbury

Jan Sedivy

Mukund Padmanabhan

David C Haws

Ruhi Sarikaya

Dimitrios B Dimitriadis

Paul C Tang

Dan Jiang

Fu-Hua Liu

Zoltan Tueske

Yinghui Huang

Ramesh Ambat Gopinath

Maria Elena Smith

Abraham P Ittycheriah

Lidia Mangu

Peder Andreas Olsen

Michael Daniel Monkowski

Edward Adam Epstein

William Stuart Meisel

Charles T Rutherfoord

Arthur J Nadas

Rick Hamilton

Hakan Erdogan

Yuging Gao

Ridley M Whitaker

Rick A Hamilton, Ii

Masayuki A Suzuki

Chandrasekhar Narayanaswami

Wlodek Wlodzimierz Zadrozny

Liam David Comerford

Harry William Printz

Karthik Visweswariah

Sreeram Viswanath Balakrishnan

Andrew Aaron

Salim Estephan Roukos

Stanley S Chen

Benoit Maison

Michelle Yoonkyung Kim

Peter F Brown

John M Lucassen

Zhi Wei Shuang

Peter Thomas Brunet

Stephen Graham Lawrence

Geoffrey G Zweig

Raul Fernandez

Laurence P Sansone

Catalina M Danis

Hamed A Ellozy

Steven J Friedland

Peter V Desouza

Hari Chittaluru

Scott Elliot Axelrod

Yongxin Li

Satya Dharanipragada

David Nahomoo

Latit R Bahl

P S Gopalakrishnan

Peter D De Souza

Lahit R Bahl

Karen Livescu

Shane Settle

Embodiments of the present invention provide computer implemented methods, computer program products and computer systems. For example, embodiments of the present invention can access one or more intents and associated entities from limited amount of speech to text training data in a single language. Embodiments of the present invention can locate speech to text training data in one or more other languages using the accessed one or more intents and associated entities to locate speech to text training data in the one or more other languages different than the single language. Embodiments of the present invention can then train a neural network based on the limited amount of speech to text training data in the single language and the located speech to text training data in the one or more other languages.

Multilingual intent recognition

An illustrative embodiment includes a method for training an end-to-end (E2E) spoken language understanding (SLU) system. The method includes receiving a training corpus comprising a set of text classified using one or more sets of semantic labels but unpaired with speech and using the set of unpaired text to train the E2E SLU system to classify speech using at least one of the one or more sets of semantic labels. The method may include training a text-to-intent model using the set of unpaired text; and training a speech-to-intent model using the text-to-intent model. Alternatively or additionally, the method may include using a text-to-speech (TTS) system to generate synthetic speech from the unpaired text; and training the E2E SLU system using the synthetic speech.

Leveraging unpaired text data for training end-to-end spoken language understanding systems

Method, apparatus, and computer program product are provided for customizing an automatic closed captioning system. In some embodiments, at a data use (DU) location, an automatic closed captioning system that includes a base model is provided, search criteria are defined to request from one or more data collection (DC) locations, a search request based on the search criteria is sent to the one or more DC locations, relevant closed caption data from the one or more DC locations are received responsive to the search request, the received relevant closed caption data are processed by computing a confidence score for each of a plurality of data sub-sets of the received relevant closed caption data and selecting one or more of the data sub-sets based on the confidence scores, and the automatic closed captioning system is customized by using the selected one or more data sub-sets to train the base model.

Using closed captions as parallel training data for customization of closed captioning systems

Aspects of the present disclosure describe techniques for identifying and recovering out-of-vocabulary words in transcripts of a voice data recording using word recognition models and word sub-unit recognition models. An example method generally includes receiving a voice data recording for transcription into a textual representation of the voice data recording. The voice data recording is transcribed into the textual representation using a word recognition model. An unknown word is identified in the textual representation, and the unknown word is reconstructed based on recognition of sub-units of the unknown word generated by a sub-unit recognition model. The textual representation of the voice data recording is modified by replacing the unknown word with the reconstruction of the unknown word, and the modified textual representation is output.

Detecting and recovering out-of-vocabulary words in voice-to-text transcription systems

In an approach to soft-forgetting training, one or more computer processors train a first model utilizing one or more training batches wherein each training batch of the one or more training batches comprises one or more blocks of information. The one or more computer processors, responsive to a completion of the training of the first model, initiate a training of a second model utilizing the one or more training batches. The one or more computer processors jitter a random block size for each block of information for each of the one or more training batches for the second model. The one or more computer processors unroll the second model over one or more non-overlapping contiguous jittered blocks of information. The one or more computer processors, responsive to the unrolling of the second model, reduce overfitting for the second model by applying twin regularization.

Soft-forgetting for connectionist temporal classification based automatic speech recognition

A method, a system, and a computer program product are provided. Speech signals from a medical conversation between a medical provider and a patient are converted to text based on a first domain model associated with a medical scenario. The first domain model is selected from multiple domain models associated with a workflow of the medical provider. One or more triggers are detected, each of which indicates a respective change in the medical scenario. A corresponding second domain model is applied to the medical conversation to more accurately convert the speech signals to text in response to each of the detected one or more triggers. The corresponding second domain model is associated with a respective change in the medical scenario of the workflow of the medical provider. A clinical note is provided based on the text produced by converting the speech signals.

Optimizing speech to text conversion and text summarization using a medical provider workflow model

Audio features, such as perceptual linear prediction (PLP) features and time derivatives thereof, are extracted from frames of training audio data including speech by multiple speakers, and silence, such as by using linear discriminant analysis (LDA). The frames are clustered into k-means clusters using distance measures, such as Mahalanobis distance measures, of means and variances of the extracted audio features of the frames. A recurrent neural network (RNN) is trained on the extracted audio features of the frames and cluster identifiers of the k-means clusters into which the frames have been clustered. The RNN is applied to audio data to segment audio data into segments that each correspond to one of the cluster identifiers. Each segment can be assigned a label corresponding to one of the cluster identifiers. Speech recognition can be performed on the segments.

Using recurrent neural network for partitioning of audio data into segments that each correspond to a speech feature cluster identifier

A method (and structure and computer product) for learning Out-of-Vocabulary (OOV) words in an Automatic Speech Recognition (ASR) system includes using an Acoustic Word Embedding Recurrent Neural Network (AWE RNN) to receive a character sequence for a new OOV word for the ASR system, the RNN providing an Acoustic Word Embedding (AWE) vector as an output thereof. The AWE vector output from the AWE RNN is provided as an input into an Acoustic Word Embedding-to-Acoustic-to-Word Neural Network (AWE&#x2192;A2W NN) trained to provide an OOV word weight value from the AWE vector. The OOV word weight is inserted into a listing of Acoustic-to-Word (A2W) word embeddings used by the ASR system to output recognized words from an input of speech acoustic features, wherein the OOV word weight is inserted into the A2W word embeddings list relative to existing weights in the A2W word embeddings list.

Recognition of out-of-vocabulary in direct acoustics-to-word speech recognition using acoustic word embedding

A method, computer system, and a computer program product for optimizing speech recognition in a smart medical room. The present invention may include selecting, from a database, one or more speech domain models based on a plurality of signals from a plurality of biometric sensors associated with a plurality of medical equipment, wherein the one or more speech domain models are trained with one or more feedback from a clinician based on a medical encounter and from a continuous feedback display in the smart medical room, wherein the one or more feedback from the clinician is based on an optional notification to the clinician to confirm the one or more speech models in use.

Smart medical room optimization of speech recognition systems

A method, computer system, and a computer program product for optimizing speech recognition in a smart medical room. The present invention may include receiving a piece of verbal data associated with a medical encounter from one or more audio recording devices. The present invention may also include accessing a plurality of signals from a plurality of biometric sensors associated with a plurality of medical equipment associated with the smart medical room based on the received piece of verbal data associated with the medical encounter. The present invention may further include selecting, from a database, one or more speech domain models based on the accessed plurality of signals from the plurality of biometric sensors associated with the plurality of medical equipment, wherein the one or more speech domain models are utilized to optimize a transcription of speech during the medical encounter in the smart medical room.

Speaker diarization is performed on audio data including speech by a first speaker, speech by a second speaker, and silence. The speaker diarization includes segmenting the audio data using a long short-term memory (LSTM) recurrent neural network (RNN) to identify change points of the audio data that divide the audio data into segments. The speaker diarization includes assigning a label selected from a group of labels to each segment of the audio data using the LSTM RNN. The group of labels comprising includes labels corresponding to the first speaker, the second speaker, and the silence. Each change point is a transition from one of the first speaker, the second speaker, and the silence to a different one of the first speaker, the second speaker, and the silence. Speech recognition can be performed on the segments that each correspond to one of the first speaker and the second speaker.

Using long short-term memory recurrent neural network for speaker diarization segmentation

A method for spoken term detection, comprising generating a time-marked word list, wherein the time-marked word list is an output of an automatic speech recognition system, generating an index from the time-marked word list, wherein generating the index comprises creating a word loop weighted finite state transducer for each utterance, i, receiving a plurality of keyword queries, and searching the index for a plurality of keyword hits.

Method and system for order-free spoken term detection

Techniques for converting spoken speech into written speech are provided. The techniques include transcribing input speech via speech recognition, mapping each spoken utterance from input speech into a corresponding formal utterance, and mapping each formal utterance into a stylistically formatted written utterance.

Translating between spoken and written language

Growing community of inventors

White Plains, NY, United States of America

Michael Alan Picheny