Huawei Technologies Co., Limited

Md Akmal Haidar

Mehdi Rezagholizadeh

Dalei Wu

Alan Do-Omri

Vahid Partovi Nia

Chao Xing

Pascal Poupart

Ahmad Rashid

Methods, devices and processor-readable media for knowledge distillation using intermediate representations are described. A student model is trained using a Dropout-KD approach in which intermediate layer selection is performed efficiently such that the skip, search, and overfitting problems in intermediate layer KD may be solved. Teacher intermediate layers are selected randomly at each training epoch, with the layer order preserved to avoid breaking information flow. Over the course of multiple training epochs, all of the teacher intermediate layers are used for knowledge distillation. A min-max data augmentation method is also described based on the intermediate layer selection of the Dropout-KD training method.

Methods, devices and media for improving knowledge distillation using intermediate representations

Computer implemented method and system for automatic speech recognition. A first speech sequence is processed, using a time reduction operation of an encoder NN, into a second speech sequence comprising a second set of speech frame feature vectors that each concatenate information from a respective plurality of speech frame feature vectors included in the first set and includes fewer speech frame feature vectors than the first speech sequence. The second speech sequence is transformed, using a self-attention operation of the encoder NN, into a third speech sequence comprising a third set of speech frame feature vectors. The third speech sequence is processed using a probability operation of the encoder NN, to predict a sequence of first labels corresponding to the third set of speech frame feature vectors, and using a decoder NN to predict a sequence of second labels corresponding to the third set of speech frame feature vectors.

Transformer-based automatic speech recognition system incorporating time-reduction layer

According to embodiments, an encoder neural network receives a one-hot representation of a real text. The encoder neural network outputs a latent representation of the real text. A decoder neural network receives random noise data or artificial code generated by a generator neural network from random noise data. The decoder neural network outputs softmax representation of artificial text. The decoder neural network receives the latent representation of the real text. The decoder neural network outputs a reconstructed softmax representation of the real text. A hybrid discriminator neural network receives a first combination of the soft-text and the latent representation of the real text and a second combination of the softmax representation of artificial text and the artificial code. The hybrid discriminator neural network outputs a probability indicating whether the second combination is similar to the first combination. Additional embodiments for utilizing latent representation are also disclosed.

Latent space and text-based generative adversarial networks (LATEXT-GANs) for text generation

A method and machine translation system for bi-directional translation of textual sequences between a first language and a second language are described. The machine translation system includes a first autoencoder configured to receive a vector representation of a first textual sequence in the first language and encode the vector representation of the first textual sequence into a first sentence embedding. The machine translation system also includes a sum-product network (SPN) configured to receive the first sentence embedding and generate a second sentence embedding by maximizing a first conditional probability of the second sentence embedding given the first sentence embedding and a second autoencoder receiving the second sentence embedding, the second autoencoder being trained to decode the second sentence embedding into a vector representation of a second textual sequence in the second language.

System and method for bi-directional translation using sum-product networks

In accordance to embodiments, an encoder neural network is configured to receive a one-hot representation of a real text and output a latent representation of the real text generated from the one-hot representation of the real text. A decoder neural network is configured to receive the latent representation of the real text, and output a reconstructed softmax representation of the real text from the latent representation of the real text, the reconstructed softmax representation of the real text is a soft-text. A generator neural network is configured to generate artificial text based on random noise data. A discriminator neural network is configured to receive the soft-text and receive a softmax representation of the artificial text, and output a probability indicating whether the softmax representation of the artificial text received by the discriminator neural network is not from the generator neural network.

Autoencoder-based generative adversarial networks for text generation

In at least one broad aspect, described herein are systems and methods in which a latent representation shared between two languages is built and/or accessed, and then leveraged for the purpose of text generation in both languages. Neural text generation techniques are applied to facilitate text generation, and in particular the generation of sentences (i.e., sequences of words or subwords) in both languages, in at least some embodiments.

Systems and methods for multilingual text generation field

A method and system for augmenting a training dataset for a generative adversarial network (GAN). The training dataset includes labelled data samples and unlabelled data samples. The method includes: receiving generated samples generated using a first neural network of the GAN and the unlabelled samples of training dataset; determining a decision value for a sample from a decision function, wherein the sample is a generated sample of the generated samples or an unlabelled sample of the unlabelled samples of the training dataset; comparing the decision value to a threshold; in response to determining that the decision value exceeds the threshold: predicting a label for a sample; assigning the label to the sample; and augmenting the training dataset to include the sample with the assigned label as a labelled sample.

Self-training method and system for semi-supervised learning with generative adversarial networks

Method and system for performing semi-supervised regression with a generative adversarial network (GAN) that includes a generator comprising a first neural network and a discriminator comprising a second neural network, comprising: outputting, from the first neural network, generated samples derived from a random noise vector; inputting, to the second neural network, the generated samples, a plurality of labelled training samples, and a plurality of unlabelled training samples; and outputting, from the second neural network, a predicted continuous label for each of a plurality of the generated samples and unlabelled samples.

Growing community of inventors

Montreal, Canada

Md Akmal Haidar