Extracting explanations from attention-based models

Abstract

Providing an explanation for model outcome can include receiving input data, and passing the input data through an attention-based neural network, where the attention-based neural network learns attention weights associated with contextual embeddings corresponding to tokens of the input data and predicts an outcome corresponding to the input data. Based on an attention weight associated with a contextual embedding corresponding to a token of the input data, a signed relevance score can be determined to associate with the token for quantifying the token's relevance to the outcome. Based on the signed relevance score, an explanation of the token's contribution toward or against the outcome can be provided. The signed relevance score can be computed as a gradient of loss with respect to the attention weight.