Training speech synthesis to generate distinct speech sounds

Abstract

A method () of training a text-to-speech (TTS) model () includes obtaining training data () including reference input text () that includes a sequence of characters, a sequence of reference audio features () representative of the sequence of characters, and a sequence of reference phone labels () representative of distinct speech sounds of the reference audio features. For each of a plurality of time steps, the method includes generating a corresponding predicted audio feature () based on a respective portion of the reference input text for the time step and generating, using a phone label mapping network (), a corresponding predicted phone label () associated with the predicted audio feature. The method also includes aligning the predicted phone label with the reference phone label to determine a corresponding predicted phone label loss () and updating the TTS model based on the corresponding predicted phone label loss.