Using hierarchical mixture-of-experts language models with microphone fidelity routing

Abstract

Implementations relate to leveraging a generative model to process a transcript of user speech and a spectrogram that reflects audible features of audio data capturing the user speech, to generate a response. The generative model can include a Mixture-of-Experts (MoE) subnetwork and a gating subnetwork. The MoE subnetwork can include multiple expert models trained (e.g., fine-tuned) respectively based on content recognized from audio data corresponding to different ranges of signal-to-noise ratios. The gating subnetwork can be trained to select a subset of expert models from the multiple expert models based on the spectrogram that reflects the audible features (e.g., SNR ratios) of the audio data. The gating subnetwork can further be used to determine a gating weight for each expert model from the selected subset, and expert output from each can be combined with a corresponding gating weight, to generate a MoE layer output that reflects a response.