Microsoft Technology Licensing, LLC

Jian Luan

Max Leung

Lei He

Yu Shi

Henry Gabryjelski

Mei-Yuh Hwang

Kaisheng N Yao

Dapeng Li

Bo Yan

Pei Zhao

Malone Ma

Zhen Liu

Sheng Zhao

Furu Wei

Qinying Liao

Nan Yang

Xiang Xu

Linfang Wang

Daniela Braga

Shihui Liu

Yahuan Cong

Ran Zhang

Zhe Xiao

Chi Xiu

Xingyu Na

Jianzhong Ju

Hairong Xia

The present disclosure provides methods and apparatuses for phrase-based end-to-end text-to-speech (TTS) synthesis. A text may be obtained. A target phrase in the text may be identified. A phrase context of the target phrase may be determined. An acoustic feature corresponding to the target phrase may be generated based at least on the target phrase and the phrase context. A speech waveform corresponding to the target phrase may be generated based on the acoustic feature.

Phrase-based end-to-end text-to-speech (TTS) synthesis

The present disclosure provides method and apparatus for generating a response in a human-machine conversation. A first sound input may be received in the conversation. A first audio attribute may be extracted from the first sound input, wherein the first audio attribute indicates a first condition of a user. A second sound input may be received in the conversation. A second audio attribute may be extracted from the second sound input, wherein the second audio attribute indicates a second condition of a user. A difference between the second audio attribute and the first audio attribute is determined, wherein the difference indicates a condition change of the user from the first condition to the second condition. A response to the second sound input is generated based at least on the condition change.

Generating response in conversation

A method and system for automatic dubbing method is disclosed, comprising, responsive to receiving a selection of media content for playback on a user device by a user of the user device, processing extracted speeches of a first voice from the media content to generate replacement speeches using a set of phenomes of a second voice of the user of the user device, and replacing the extracted speeches of the first voice with the generated replacement speeches in the audio portion of the media content for playback on the user device.

Automatic dubbing method and apparatus

An example intent-recognition system comprises a processor and memory storing instructions. The instructions cause the processor to receive speech input comprising spoken words. The instructions cause the processor to generate text results based on the speech input and generate acoustic feature annotations based on the speech input. The instructions also cause the processor to apply an intent model to the text result and the acoustic feature annotations to recognize an intent based on the speech input. An example system for adapting an emotional text-to-speech model comprises a processor and memory. The memory stores instructions that cause the processor to receive training examples comprising speech input and receive labelling data comprising emotion information associated with the speech input. The instructions also cause the processor to extract audio signal vectors from the training examples and generate an emotion-adapted voice font model based on the audio signal vectors and the labelling data.

Intent recognition and emotional text-to-speech learning

An automatic dubbing method is disclosed. The method comprises: extracting speeches of a voice from an audio portion of a media content (); obtaining a voice print model for the extracted speeches of the voice (); processing the extracted speeches by utilizing the voice print model to generate replacement speeches (); and replacing the extracted speeches of the voice with the generated replacement speeches in the audio portion of the media content ().

The present disclosure provides a technical solution of highly empathetic TTS processing, which not only takes a semantic feature and a linguistic feature into consideration, but also assigns a sentence ID to each sentence in a training text to distinguish sentences in the training text. Such sentence IDs may be introduced as training features into a processing of training a machine learning model, so as to enable the machine learning model to learn a changing rule for the changing of acoustic codes of sentences with a context of sentence. A speech naturally changed in rhythm and tone may be output to make TTS more empathetic by performing TTS processing with the trained model. A highly empathetic audio book may be generated using the TTS processing provided herein, and an online system for generating a highly empathetic audio book may be established with the TTS processing as a core technology.

Highly empathetic ITS processing

In accordance with implementations of the subject matter described herein, there is provided a solution for supporting a machine to automatically generate a song. In this solution, an input from a user is used to determine a creation intention of the user with respect to a song to be generated. Lyrics of the song are generated based on the creation intention. Then, a template for the song is generated based at least in part on the lyrics. The template indicates a melody matching with the lyrics. In this way, it is feasible to automatically create the melody and lyrics which not only conform to the creation intention of the user but also match with each other.

Automatic song generation

Multi-voice font interpolation is provided. A multi-voice font interpolation engine allows the production of computer generated speech with a wide variety of speaker characteristics and/or prosody by interpolating speaker characteristics and prosody from existing fonts. Using prediction models from multiple voice fonts, the multi-voice font interpolation engine predicts values for the parameters that influence speaker characteristics and/or prosody for the phoneme sequence obtained from the text to spoken. For each parameter, additional parameter values are generated by a weighted interpolation from the predicted values. Modifying an existing voice font with the interpolated parameters changes the style and/or emotion of the speech while retaining the base sound qualities of the original voice. The multi-voice font interpolation engine allows the speaker characteristics and/or prosody to be transplanted from one voice font to another or entirely new speaker characteristics and/or prosody to be generated for an existing voice font.

Voice font speaker and prosody interpolation

Techniques for converting text to speech having emotional content. In an aspect, an emotionally neutral acoustic trajectory is predicted for a script using a neutral model, and an emotion-specific acoustic trajectory adjustment is independently predicted using an emotion-specific model. The neutral trajectory and emotion-specific adjustments are combined to generate a transformed speech output having emotional content. In another aspect, state parameters of a statistical parametric model for neutral voice are transformed by emotion-specific factors that vary across contexts and states. The emotion-dependent adjustment factors may be clustered and stored using an emotion-specific decision tree or other clustering scheme distinct from a decision tree used for the neutral voice model.

Text-to-speech with emotional content

Dynamic features are utilized with CRFs to handle long-distance dependencies of output labels. The dynamic features present a probability distribution involved in explicit distance from/to a special output label that is pre-defined according to each application scenario. Besides the number of units in the segment (from the previous special output label to the current unit), the dynamic features may also include the sum of any basic features of units in the segment. Since the added dynamic features are involved in the distance from the previous specific label, the searching lattice associated with Viterbi searching is expanded to distinguish the nodes with various distances. The dynamic features may be used in a variety of different applications, such as Natural Language Processing, Text-To-Speech and Automatic Speech Recognition. For example, the dynamic features may be used to assist in prosodic break and pause prediction.

Growing community of inventors

Beijing, China

Jian Luan