Training natural language processing network based on existing network

Abstract

Some embodiments provide a method for training a first NLP network based on a previously-trained second NLP network. The method propagates text inputs through the first network to generate a first set of output vectors and the second network to generate a second set of output vectors. Each first-set vector generated based on a text input has a corresponding second-set vector generated based on the same text input and each vector of the first and second sets has a same number of vector components. The method computes a value for a loss function that emphasizes a maximum disparity between components of first-set vectors and corresponding components of second-set vectors. The method trains the first network using the computed loss function value to minimize the maximum disparity between the first-set vector components and corresponding second-set vector components so that the first network produces outputs similar to outputs of the second network.