Generating expressive facial animation data from speech audio using speech emotion recognition

Abstract

This specification provides a computer-implemented method for generating facial animation data. The facial animation data animates a face in a video game in accordance with speech sounds and speech emotion of speech audio. Speech audio data representing the speech audio is processed using a machine-learned speech emotion recognition model. One or more emotion representations that represent the emotion of the speech audio are generated. For each of the one or more emotion representations, a respective conditioning input for a facial animation generative model is determined. Facial animation data for the speech audio data is generated, comprising processing, using the facial animation generative model: (i) input data derived from the speech audio data, and (ii) the one or more conditioning inputs.