Deep learning optimizer for fine-tuning while dynamically mitigating catastrophic forgetting

Abstract

Disclosed are embodiments for a deep learning optimizer for fine-tuning while dynamically mitigating catastrophic forgetting. In some aspects, a method includes saving an original state of a model to a buffer, the original state comprising original weights of the model; apply new gradients to intermediate model states of the model during at least one intermediate training step of training of the model; calculate, at a final training step of the training, a displacement vector representing a difference between current weights of the model and the original weights of the model; responsive to the displacement vector and a final gradient of the final training step being inconsistent in terms of gradient direction, perform an orthogonal projection of the final gradient on the displacement vector to generate an orthogonal gradient; and applying the orthogonal gradient to the original state of the model at the final training step of the training.