Amazon Technologies, Inc.

Srikanth Ronanki

Thomas Renaud Drugman

Viacheslav Klimkov

Andrew Paul Breen

Thomas Edward Merritt

Jaime Lorenzo Trueba

Roberto Barra-Chicote

Sravan Babu Bodapati

Katrin Kirchhoff

Alexander Galkin

Monica Lakshmi Sunkara

Dhanush Bekal Kannangola

During text-to-speech processing, a speech model creates output audio data, including speech, that corresponds to input text data that includes a representation of the speech. A spectrogram estimator estimates a frequency spectrogram of the speech; the corresponding frequency-spectrogram data is used to condition the speech model. A plurality of acoustic features corresponding to different segments of the input text data, such as phonemes, syllable-level features, and/or word-level features, may be separately encoded into context vectors; the spectrogram estimator uses these separate context vectors to create the frequency spectrogram.

Text-to-speech (TTS) processing

Techniques for predicting punctuation and casing using multimodal fusion are described. An exemplary method includes processing generated text by: tokenizing the generated text into sub-words, and generating a sequence of lexical features for the sub-words using a pre-trained lexical encoder; processing audio of the audio by: generating a sequence of frame level acoustic embeddings using a pre-trained acoustic encoder on the audio, and generating task specific embeddings from the frame level acoustic embeddings; performing multimodal fusion of the sub-word level acoustic embeddings and the sequence of lexical features by: aligning the task specific embeddings to the sequence of lexical features, and combining the sequence of lexical features and aligned acoustic sequence; predicting punctuation and casing from the combined sequence of lexical features and aligned acoustic sequence; concatenating the sub-words of the text, and applying the predicted punctuation and casing; and outputting text having the predicted punctuation and casing.

Multimodal based punctuation and/or casing prediction

Audio data from a first, source speaker is received and processed to determine linguistic units and vocal characteristics corresponding to those linguistic units. The linguistic units may either be determined from received text data or may be determined from the audio data using automatic speech recognition. A model is trained using training data from a second, target speaker. The trained model concatenates the linguistic units with the vocal characteristics to produce output speech that has the 'voice' of the target speaker and the vocal characteristics of the source speaker.

Text-to-speech (TTS) processing with transfer of vocal characteristics

Growing community of inventors

Cambridge, United Kingdom

Srikanth Ronanki