Methods and systems of noise aware audio visual speech denoising

Abstract

The present subject matter provides a method for de-noising an audio visual speech. The method includes modeling a noise in the audio visual speech using a noisy speech from audio data associated with the audio visual speech to generate a reconstructed noise signal. The method includes estimating the reconstructed noise signal in the audio visual speech using an audio signal and a plurality of visual frames. The method includes partitioning the reconstructed noise signal into a plurality of windows and calculate an energy associated with each window. The method includes estimating a noise strength in each window by performing a soft max operation to obtain one or more refined audio features. The method includes fusing the one or more refined audio features and one or more visual features using the noise strength to generate an output that is passed through a decoder to obtain a de-noised audio visual speech.