Amazon Technologies, Inc.

Monica Lakshmi Sunkara

Sravan Babu Bodapati

Katrin Kirchhoff

Srikanth Ronanki

Ashish Vishwanath Shenoy

Jeffrey John Farris

Nilaksh Das

Mohit Gupta

John Baker

Ganesh Kumar Gella

Harshal Pimpalkhute

Varun Sembium Varadarajan

Anubhav Mishra

Deepthi Devaiah Devanira

Vivek Govindan

Silviu Mihai Burz

Yide Zou

Dhanush Bekal Kannangola

Jinglun Cai

Nicholas G Aldridge

Devang Kulshreshtha

Chaitanya Shivade

Language models may be dynamically updated for trending entities based on tuning data for particular users. A user may provide specific tuning data associated with trending entities within a class to generate a weight map for a language model. A class based model may be trained using the weight map specific for the user for the trending entities. Additionally, weights may be further boosted using a boosting language model to emphasize the trending entities.

Dynamic language model updates with boosting

Systems and methods are described for converting speech to text. In one aspect, a method for converting speech to text includes generating an initial transcript of an audio or video file. At least one named entity may be extracted from the initial transcript using entity recognition. A subset of nodes of a knowledge graph that include the at least one named entity may be selected, where the subset of nodes of the knowledge graph correspond to a set of named entities. The method may further include encoding the set of named entities to generate a set of entity embeddings. The speech in the audio or video file may then be decoded using the set of entity embeddings to produce a final transcript of the audio file.

Infusing knowledge graphs into automatic speech recognition

Bias may be removed from automatic speech recognition model predictions using internal language model estimates. Audio data may be received for speech recognition. The audio data may be processed both through an automatic speech recognition model to produce original word token predictions and masked in different portions of the audio data to produce other word token predictions for the masked audio. A comparison of the original word token predictions and the other word token predictions may provide an estimate of an internal language model for the automatic speech recognition model. This estimate can be used to modify the original word token predictions to remove the lexical bias and produce a speech prediction.

Removing bias from automatic speech recognition models using internal language model estimates

Techniques for at least the generation of a chatbot built from a custom vocabulary and to use runtime hints during inference are described. In some examples, the generation of the chatbot includes receiving a request to build a chatbot using a bot definition and a custom vocabulary, wherein the chatbot is to use runtime hints during usage; building the chatbot from the bot definition and custom vocabulary by at least: generating automatic speech recognition (ASR) artifacts to be used in decoding audio input into the chatbot into text for at least one other component of the chatbot to use in determining a next act to be performed, the ASR artifacts including artifacts that use the custom vocabulary and artifacts that do not use the custom vocabulary, and storing the ASR artifacts.

Dynamically selectable automated speech recognition using a custom vocabulary

Techniques for personalized batch and streaming speech-to-text transcription of audio reduce the error rate of automatic speech recognition (ASR) systems in transcribing rare and out-of-vocabulary words. The techniques achieve personalization of connectionist temporal classification (CT) models by using adaptive boosting to perform biasing at the level of sub-words. In addition to boosting, the techniques encompass a phone alignment network to bias sub-word predictions towards rare long-tail words and out-of-vocabulary words. A technical benefit of the techniques is that the accuracy of speech-to-text transcription of rare and out-of-vocabulary words in a custom vocabulary by automatic speech recognition (ASR) system can be improved without having to train the ASR system on the custom vocabulary. Instead, the techniques allow the same ASR system trained on a base vocabulary to realize the accuracy improvements for different custom vocabularies spanning different domains.

Personalized batch and streaming speech-to-text transcription of audio

Portions of text data generated from inverse text normalization may be redacted. Text data for redaction may be obtained. One or more inverse text normalization models may be applied to the text data to generate normalized text data. A machine learning model, trained to recognize text for redaction, may be applied to identify portions of the normalized text data for redaction. The identified portions may be redacted and the redacted normalized text provided to a destination.

Redacting portions of text transcriptions generated from inverse text normalization

Techniques for predicting punctuation and casing using multimodal fusion are described. An exemplary method includes processing generated text by: tokenizing the generated text into sub-words, and generating a sequence of lexical features for the sub-words using a pre-trained lexical encoder; processing audio of the audio by: generating a sequence of frame level acoustic embeddings using a pre-trained acoustic encoder on the audio, and generating task specific embeddings from the frame level acoustic embeddings; performing multimodal fusion of the sub-word level acoustic embeddings and the sequence of lexical features by: aligning the task specific embeddings to the sequence of lexical features, and combining the sequence of lexical features and aligned acoustic sequence; predicting punctuation and casing from the combined sequence of lexical features and aligned acoustic sequence; concatenating the sub-words of the text, and applying the predicted punctuation and casing; and outputting text having the predicted punctuation and casing.

Multimodal based punctuation and/or casing prediction

Growing community of inventors

San Jose, CA, United States of America

Monica Lakshmi Sunkara