Nanjing Silicon Intelligence Technology Co., Ltd.

Zheng Liao

Huapeng Sima

Cuicui Tang

This application provides a audio-driven three-dimensional facial animation model generation method and apparatus, and an electronic device. The method includes: acquiring sample data including sample audio data, sample speaking style data, and a sample blend shape value; performing feature extraction on the sample audio data to obtain a sample audio feature; performing convolution on the sample audio feature based on a to-be-trained audio-driven three-dimensional facial animation model to obtain an initial audio feature, and performing encoding on the sample speaking style data based on the to-be-trained audio-driven three-dimensional facial animation model to obtain a sample speaking style feature; performing encoding on the initial audio feature and the sample speaking style feature based on the to-be-trained audio-driven three-dimensional facial animation model, to obtain an output blend shape value; and performing calculation on the sample blend shape value and the output blend shape value to obtain a loss function value.

Audio-driven three-dimensional facial animation model generation method and apparatus, and electronic device

Embodiments of the present disclosure provide a method for audio-driven character lip sync, a model for audio-driven character lip sync, and a training method therefor. A target dynamic image is obtained by acquiring a character image of a target character and speech for generating a target dynamic image, processing the character image and the speech as image-audio data that may be trained, respectively, and mixing the image-audio data with auxiliary data for training. When a large amount of sample data needs to be obtained for training in different scenarios, a video when another character speaks is used as an auxiliary video for processing, so as to obtain the auxiliary data. The auxiliary data, which replaces non-general sample data, and other data are input into a model in a preset ratio for training. The auxiliary data may improve a process of training a synthetic lip sync action of the model, so that there are no parts unrelated to the synthetic lip sync action during the training process. In this way, a problem that a large amount of sample data is required during the training process is resolved.

Method for audio-driven character lip sync, model for audio-driven character lip sync and training method therefor

A method for outputting a blend shape value includes: performing feature extraction on obtained target audio data to obtain a target audio feature vector; inputting the target audio feature vector and a target identifier into an audio-driven animation model; inputting the target audio feature vector into an audio encoding layer, determining an input feature vector of a next layer at a (2t−n)/2 time point based on an input feature vector of a previous layer between a t time point and a t−n time point, determining a feature vector having a causal relationship with the input feature vector of the previous layer as a valid feature vector, outputting sequentially target-audio encoding features, and inputting the target identifier into a one-hot encoding layer for binary vector encoding to obtain a target-identifier encoding feature; and outputting a blend shape value corresponding to the target audio data.

Growing community of inventors

Jiangsu, China

Zheng Liao