Machine learning-based approach for audio-driven avatar animation or other functions

Abstract

A method includes obtaining, using at least one processing device of an electronic device, an audio input associated with a speaker. The method also includes extracting, using a feature extractor of a trained machine learning model, audio features from the audio input. The method further includes generating (i) one or more content parameter predictions using content embeddings extracted by a content encoder and decoded by a content decoder of the trained machine learning model and (ii) one or more style parameter predictions using style embeddings extracted by a style encoder and decoded by a style decoder of the trained machine learning model. The content embeddings and the style embeddings are based on the audio features of the audio input. The trained machine learning model is trained to generate the one or more content parameter predictions and the one or more style parameter predictions using disentangled content and style embeddings.