Baidu USA LLC

Baidu.com Times Technology (beijing) Co. Ltd.

Baldu USA LLC

Dingcheng Li

Ping Li

Jingyuan Zhang

Shaogang Ren

Xu Liang Li

Hongliang Fei

Ming Wang

Zhixin Zhou

Xu Li

Siamak Zamani Dadaneh

Haiyan Yin

Xiao Huang

Generative adversarial models have several benefits; however, due to mode collapse, these generators face a quality-diversity trade-off (i.e., the generator models sacrifice generation diversity for increased generation quality). Presented herein are embodiments that improve the performance of adversarial content generation by decelerating mode collapse. In one or more embodiments, a cooperative training paradigm is employed where a second model is cooperatively trained with the generator and helps efficiently shape the data distribution of the generator against mode collapse. Moreover, embodiments of a meta learning mechanism may be used, where the cooperative update to the generator serves as a high-level meta task and which helps ensures the generator parameters after the adversarial update stay resistant against mode collapse. In experiments, tested employments demonstrated efficient slowdown of mode collapse for the adversarial text generators. Overall, embodiments outperformed the baseline approaches with significant margins in terms of both generation quality and diversity.

Meta cooperative training paradigms

Described herein are embodiments of a framework named decomposable variational autoencoder (DecVAE) to disentangle syntax and semantics by using total correlation penalties of Kullback-Leibler (KL) divergences. KL divergence term of the original VAE is decomposed such that the hidden variables generated may be separated in a clear-cut and interpretable way. Embodiments of DecVAE models are evaluated on various semantic similarity and syntactic similarity datasets. Experimental results show that embodiments of DecVAE models achieve state-of-the-art (SOTA) performance in disentanglement between syntactic and semantic representations.

Disentangle syntax and semantics in sentence representation with decomposable variational autoencoder

Described herein are embodiments of a reinforcement learning based large-scale multi-objective ranking system. Embodiments of the system may be used for optimizing short-video recommendation on a video sharing platform. Multiple competing ranking objective and implicit selection bias in user feedback are the main challenges in real-world platform. In order to address those challenges, multi-gate mixture of experts (MMoE) and soft actor critic (SAC) are integrated together into a MMoE_SAC system. Experiment results demonstrate that embodiments of the MMoE_SAC system may greatly reduce a loss function compared to systems only based on single strategies.

Video recommendation with multi-gate mixture of experts soft actor critic

Learning disentangled representations is an important topic in machine learning for a wide range of applications. Disentangled latent variables represent interpretable semantic information and reflect separate factors of variation in data. Although generative models may learn latent representations and generate data samples as well, existing models may ignore the structural information among latent representations. Described in the present disclosure are embodiments to learn disentangled latent structural representations from data using decomposable variational auto-encoders, which simultaneously learn component representations and encode component relationships. Embodiments of a novel structural prior for latent representations are disclosed to capture interactions among different data components. Embodiments are applied to data segmentation and latent relation discovery among different data components. Experiments on several datasets demonstrate the utility of the present model embodiments.

Learning latent structural relations with segmentation variational autoencoders

The thriving of deep models and generative models provides approaches to model high dimensional distributions. Generative adversarial networks (GANs) can approximate data distributions and generate data samples from the learned data manifolds as well. Presented herein are embodiments to estimate the implicit likelihoods of GAN models. In one or more embodiments, a stable inverse function of the generator is learned with the help of a variance network of the generator. The local variance of the sample distribution may be approximated by the normalized distance in the latent space. Simulation studies and likelihood testing on data sets validate embodiments, which outperformed several baseline methods in these tasks. An embodiment was also applied to anomaly detection. Experiments show that the embodiments herein can achieve state-of-the-art anomaly detection performance.

Estimating the implicit likelihoods of generative adversarial networks

Described herein are embodiments for a deep level-wise extreme multi-label learning and classification (XMLC) framework to facilitate the semantic indexing of literatures. In one or more embodiments, the Deep Level-wise XMLC framework comprises two sequential modules, a deep level-wise multi-label learning module and a hierarchical pointer generation module. In one or more embodiments, the first module decomposes terms of domain ontology into multiple levels and builds a special convolutional neural network for each level with category-dependent dynamic max-pooling and macro F-measure based weights tuning. In one or more embodiments, the second module merges the level-wise outputs into a final summarized semantic indexing. The effectiveness of Deep Level-wise XMLC framework embodiments is demonstrated by comparing it with several state-of-the-art methods of automatic labeling on various datasets.

Systems and methods for large scale semantic indexing with deep level-wise extreme multi-label learning

Described herein are embodiments of a framework named as total correlation variational autoencoder (TC_VAE) to disentangle syntax and semantics by making use of total correlation penalties of KL divergences. One or more Kullback-Leibler (KL) divergence terms in a loss for a variational autoencoder are discomposed so that generated hidden variables may be separated. Embodiments of the TC_VAE framework were examined on semantic similarity tasks and syntactic similarity tasks. Experimental results show that better disentanglement between syntactic and semantic representations have been achieved compared with state-of-the-art (SOTA) results on the same data sets in similar settings.

Total correlation variational autoencoder strengthened with attentions for segmenting syntax and semantics

Described herein are embodiments for question answering over knowledge graph using a Knowledge Embedding based Question Answering (KEQA) framework. Instead of inferring an input questions' head entity and predicate directly, KEQA embodiments target jointly recovering the question's head entity, predicate, and tail entity representations in the KG embedding spaces. In embodiments, a joint distance metric incorporating various loss terms is used to measure distances of a predicated fact to all candidate facts. In embodiments, the fact with the minimum distance is returned as the answer. Embodiments of a joint training strategy are also disclosed for better performance. Performance evaluation on various datasets demonstrates the effectiveness of the disclosed systems and methods using the KEQA framework.

Knowledge-graph-embedding-based question answering

Leveraging domain knowledge is an effective strategy for enhancing the quality of inferred low-dimensional representations of documents by topic models. Presented herein are embodiments of a Bayesian nonparametric model that employ knowledge graph (KG) embedding in the context of topic modeling for extracting more coherent topics; embodiments of the model may be referred to as topic modeling with knowledge graph embedding (TMKGE). TMKGE embodiments are hierarchical Dirichlet process (HDP)-based models that flexibly borrow information from a KG to improve the interpretability of topics. Also, embodiments of a new, efficient online variational inference method based on a stick-breaking construction of HDP were developed for TMKGE models, making TMKGE suitable for large document corpora and KGs. Experiments on datasets illustrate the superior performance of TMKGE in terms of topic coherence and document classification accuracy, compared to state-of-the-art topic modeling methods.

Integration of knowledge graph embedding into topic modeling with hierarchical Dirichlet process

Described herein are embodiments for end-to-end reinforcement learning based coreference resolution models to directly optimize coreference evaluation metrics. Embodiments of a reinforced policy gradient model are disclosed to incorporate reward associated with a sequence of coreference linking actions. Furthermore, maximum entropy regularization may be used for adequate exploration to prevent a model embodiment from prematurely converging to a bad local optimum. Experiments on datasets compared with state-of-the-art methods verified the effectiveness of embodiments.

Systems and methods for end-to-end deep reinforcement learning based coreference resolution

Described herein are embodiments of a unified neural network framework to integrate Topic modeling, Word embedding and Entity Embedding (TWEE) for representation learning of inputs. In one or more embodiments, a novel topic sparse autoencoder is introduced to incorporate discriminative topics into the representation learning of the input. Topic distributions of inputs are generated from a global viewpoint and are utilized to enable autoencoder to learn topical representations. A sparsity constraint may be added to ensure that the most discriminative representations are related to topics. In addition, both words and entity related information may be embedded into the network to help learn a more comprehensive input representation. Extensive empirical experiments show that embodiments of the TWEE framework outperform the state-of-the-art methods on different datasets.

Representation learning for input classification via topic sparse autoencoder and entity embedding

Described herein are embodiments for systems and methods for mutual machine learning with global topic discovery and local word embedding. Both topic modeling and word embedding map documents onto a low-dimensional space, with the former clustering words into a global topic space and the latter mapping word into a local continuous embedding space. Embodiments of Topic Modeling and Sparse Autoencoder (TMSA) framework unify these two complementary patterns by constructing a mutual learning mechanism between word co-occurrence based topic modeling and autoencoder. In embodiments, word topics generated with topic modeling are passed into auto-encoder to impose topic sparsity for the autoencoder to learn topic-relevant word representations. In return, word embedding learned by autoencoder is sent back to topic modeling to improve the quality of topic generations. Performance evaluation on various datasets demonstrates the effectiveness of the disclosed TMSA framework in discovering topics and embedding words.

Growing community of inventors

Sammamish, WA, United States of America

Dingcheng Li