Language processing using a neural network

Abstract

This disclosure relates to language processing using a neural network. In one aspect, a method includes performing, at an embedding layer of a neural network, embedding processing on a current input to obtain feature vectors. The neural network includes at least one transformer layer that includes a first transformer layer including an attention layer and a pooling layer. A quantity P of input vectors are obtained at the attention layer. P intermediate vectors are determined based on the input vectors. For each input vector of the P input vectors, a respective intermediate vector is obtained using the corresponding input vector as a center and based on correlation values calculated between the input vector and each other input vector in a predetermined attention window range. The P intermediate vectors are combined to form a quantity Q of output vectors. Output vectors are generated as a feature representation of the current input.