Guided data selection for masked speech modeling based on an average score assigned to encoded representations of an utterance

Abstract

A method of guided data selection for masked speech modeling includes obtaining a sequence of encoded representations corresponding to an utterance. For each respective encoded representation, the method includes processing the respective encoded representation to generate a corresponding probability distribution over possible speech recognition hypotheses and assigning, to the respective encode representation, a confidence score as a highest probability from the corresponding probability distribution over possible speech recognition hypotheses. The method also includes selecting a set of unmasked encoded representations to mask based on the confidence scores assigned to the sequence of encoded representations. The method also includes generating a set of masked encoded representations by masking the selected set of unmasked encoded representations. Here, each masked encoded representation in the set of masked encoded representations corresponds to a respective one of the unmasked encoded representations in the selected set of unmasked encoded representations.