Adaptive off-ramp training and inference for early exits in a deep neural network

Abstract

Systems and methods are provided for training and using a deep neural network with adaptively trained off-ramps for an early exit at an intermediate representation layer. The training includes, for respective intermediate representation layers of a sequence of intermediate representation layers, predicting a label based on the training data and comparing against a correct label. The training further includes generating a confidence value associated with the predicted label. The confidence value is based on optimizing an objective function that includes a weighted entropy of a probability distribution of the likelihood, weighted based on whether previous intermediate representation layer has accurately predicted the label. Use of the weighted entropy provides the training with a focus on predicting labels that the previous intermediate representation layers has performed poorly and not labels that have existed before the intermediate representation layer being trained. Among alternative methods include a distilled twin, parallel neural network for predicting labels using adaptively trained off-ramps.