Classifying out-of-distribution data using a contrastive loss

Abstract

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for training a neural network to (i) generate accurate network outputs for a machine learning task and (ii) generate intermediate outputs that can be used to reliably classify out-of-distribution inputs. In one aspect, a method comprises: training the neural network using supervised and contrastive losses, comprising repeatedly performing operations including: obtaining first and second network inputs; processing each network input using the neural network to generate its respective network input embedding; processing the first network input using the neural network to generate a network output; and adjusting the network parameter values using supervised and contrastive loss gradients, wherein: the supervised loss is based on: (i) the network output, and (ii) a corresponding target network output; and the contrastive loss is based on at least: (i) the first network input embedding, and (ii) the second network input embedding.