Baidu USA LLC

Baidu.com Times Technology (beijing) Co. Ltd.

Baldu USA LLC

Ping Li

Shulong Tan

Hongliang Fei

Zhixin Zhou

Zhaozhuo Xu

Dingcheng Li

Weijie Zhao

Shaogang Ren

Jianwen Xie

Ming Wang

Xu Liang Li

Xu Li

Jingyuan Zhang

Tan Yu

Puxuan Yu

Yang Zhao

Junhao Zhao

Haiyan Yin

Banghu Yin

Xingxuan Zhou

Zilong Zheng

Generative adversarial models have several benefits; however, due to mode collapse, these generators face a quality-diversity trade-off (i.e., the generator models sacrifice generation diversity for increased generation quality). Presented herein are embodiments that improve the performance of adversarial content generation by decelerating mode collapse. In one or more embodiments, a cooperative training paradigm is employed where a second model is cooperatively trained with the generator and helps efficiently shape the data distribution of the generator against mode collapse. Moreover, embodiments of a meta learning mechanism may be used, where the cooperative update to the generator serves as a high-level meta task and which helps ensures the generator parameters after the adversarial update stay resistant against mode collapse. In experiments, tested employments demonstrated efficient slowdown of mode collapse for the adversarial text generators. Overall, embodiments outperformed the baseline approaches with significant margins in terms of both generation quality and diversity.

Meta cooperative training paradigms

Different from prior works that model the internal distribution of patches within an image implicitly with a top-down latent variable model (e.g., generator), embodiments explicitly represent the statistical distribution within a single image by using an energy-based generative framework, where a pyramid of energy functions, each parameterized by a bottom-up deep neural network, are used to capture the distributions of patches at different resolutions. Also, embodiments of a coarse-to-fine sequential training and sampling strategy are presented to train the model efficiently. Besides learning to generate random samples from white noise, embodiments can learn in parallel with a self-supervised task (e.g., recover an input image from its corrupted version), which can further improve the descriptive power of the learned model. Embodiments does not require an auxiliary model (e.g., discriminator) to assist the training, and embodiments also unify internal statistics learning and image generation in a single framework.

Training energy-based models from a single image for internal learning and inference using trained models

Current pretrained vision-language models for cross-modal retrieval tasks in English depend upon on the availability of many annotated image-caption datasets for pretraining to have English text. However, the texts are not necessarily in English. Although machine translation (MT) tools may be used to translate text to English, the performance largely relies on MT's quality and may suffer from high latency problems in real-world applications. Embodiments herein address these problems by learning cross-lingual cross-modal representations for matching images and their relevant captions in multiple languages. Embodiments seamlessly combine cross-lingual pretraining objectives and cross-modal pretraining objectives in a unified framework to learn image and text in a joint embedding space from available English image-caption data, monolingual corpus, and parallel corpus. Embodiments are shown to achieve state-of-the-art performance in retrieval tasks on multimodal multilingual image caption datasets.

Systems and methods for cross-lingual cross-modal training for multimodal retrieval

Retrieval of relevant vectors produced by representation learning can critically influence the efficiency in Natural Language Processing (NLP) tasks. Presented herein are systems and methods for searching vectors via a typical nonmetric matching function: inner product. Embodiments, which construct an approximate Inner Product Delaunay Graph (IPDG) for top-1 Maximum Inner Product Search (MIPS), transform retrieving the most suitable latent vectors into a graph search problem with great benefits of efficiency. Experiments on data representations learned for different machine learning tasks verify the outperforming effectiveness and efficiency of IPDG embodiments.

Efficient retrieval of top similarity representations

Presented are systems and methods that construct BipartitE Graph INdices (BEGIN) embodiments for fast neural ranking. BEGIN embodiments comprise two types of nodes: sampled queries and base or searching objects. In one or more embodiments, edges connecting these nodes are constructed by using a neural network ranking measure. These embodiments extend traditional search-on-graph methods and lend themselves to fast neural ranking. Experimental results demonstrate the effectiveness and efficiency of such embodiments.

Fast neural ranking on bipartite graph indices

Efficient inner product search is important for many data ranking services, such as recommendation and Information Retrieval. Efficient retrieval via inner product dramatically influences the performance of such data searching and retrieval systems. To resolve deficiencies of prior approaches, embodiments of a new index graph construction approach, referred to generally as Norm Adjusted Proximity Graph (NAPG), for approximate Maximum Inner Product Search (MIPS) are presented. With adjusting factors estimated on sampled data, NAPG embodiments select more meaningful data points to connect with when constructing a graph-based index for inner product search. Extensive experiments verify that the improved graph-based index pushes the state-of-the-art of inner product search forward greatly, in the trade-off between search efficiency and effectiveness.

Norm adjusted proximity graph for fast inner product retrieval

Presented herein are embodiments of energy-based models (EBMs), which may be trained via embodiments of a multistage coarse-to-fine expanding and sampling strategy. Embodiments of the training methodology start with learning a coarse-level EBM from images at low resolution and then gradually transits to learn a finer-level EBM from images at higher resolution by expanding the energy function as the learning progresses. Embodiments are computationally efficient with smooth learning and sampling. Tested embodiments achieved the best performance on image generation amongst all EBMs and successfully synthesized high-fidelity images. Embodiments may also be used for image restoration and out-of-distribution detection. Framework embodiments may be further generalized for one-sided unsupervised image-to-image translation and beat baseline methods in terms of model size and training budget. Also presented herein are embodiments of a gradient-based generative saliency methodology to interpret the translation dynamics.

Energy-based generative models via coarse-to-fine expanding and sampling

Incremental proximity graph maintenance (IPGM) systems and methods for online ANN search support both online vertex deletion and insertion of vertices on proximity graphs. In various embodiments, updating a proximity graph comprises receiving a workload that represents a set of vertices in the proximity graph, each vertex being associated with a type of operation such as a query, insertion, or deletion. For a query or an insertion, a search may be executed on the graph to obtain a set of top-K vertices for each vertex. In the case of a deletion, a vertex may be deleted from the proximity graph, and a local or global reconnection update method may be used to reconstruct at least a portion of the proximity graph.

Proximity graph maintenance for fast online nearest neighbor search

Described herein are embodiments of a framework named decomposable variational autoencoder (DecVAE) to disentangle syntax and semantics by using total correlation penalties of Kullback-Leibler (KL) divergences. KL divergence term of the original VAE is decomposed such that the hidden variables generated may be separated in a clear-cut and interpretable way. Embodiments of DecVAE models are evaluated on various semantic similarity and syntactic similarity datasets. Experimental results show that embodiments of DecVAE models achieve state-of-the-art (SOTA) performance in disentanglement between syntactic and semantic representations.

Disentangle syntax and semantics in sentence representation with decomposable variational autoencoder

Deep neural network (DNN) models have been widely used for user-relevance content prediction. Presented herein are embodiments of a new user-relevance framework, which may be referred as Gating-Enhanced Multi-task Neural Networks (GemNN) embodiments. Neural network-based multi-task learning model embodiments herein predict user engagement with content in a coarse-to-fine manner, which gradually reduces content candidates and allows parameter sharing from upstream tasks to downstream tasks to improve the training efficiency. Also, in one or more embodiments, a gating mechanism was introduced between embedding layers and multi-layer perceptions to learn feature interactions and control the information flow fed to MLP layers. Tested embodiments demonstrated considerable improvements over prior approaches.

Systems and methods for gating-enhanced multi-task neural networks with feature interaction learning

Presented herein are embodiments of a fast search on graph methodology for Maximum Inner Product Search (MIPS). This optimization problem is challenging since traditional Approximate Nearest Neighbor (ANN) search methods may not perform efficiently in the nonmetric similarity measure. Embodiments herein are based on the property that a Möbius/Möbius-like transformation introduces an isomorphism between a subgraph of-Delaunay graph and Delaunay graph for inner product. Under this observation, embodiments of a novel graph indexing and searching methodology are presented to find the optimal solution with the largest inner product with the query. Experiments show significant improvements compared to existing methods.

Transformation for fast inner product search on graph

Described herein are system and method embodiments to improve word representation learning. Embodiments of a probabilistic prior may seamlessly integrate statistical disentanglement with word embedding. Different from previous deterministic methods, word embedding may be taken as a probabilistic generative model, and it enables imposing a prior that may identify independent factors generating word representation vectors. The probabilistic prior not only enhances the representation of word embedding, but also improves the model's robustness and stability. Furthermore, embodiments of the disclosed method may be flexibly plugged in various word embedding models. Extensive experimental results show that embodiments of the presented method may improve word representation on different tasks.

Word embedding with disentangling prior

Described herein are embodiments of a reinforcement learning based large-scale multi-objective ranking system. Embodiments of the system may be used for optimizing short-video recommendation on a video sharing platform. Multiple competing ranking objective and implicit selection bias in user feedback are the main challenges in real-world platform. In order to address those challenges, multi-gate mixture of experts (MMoE) and soft actor critic (SAC) are integrated together into a MMoE_SAC system. Experiment results demonstrate that embodiments of the MMoE_SAC system may greatly reduce a loss function compared to systems only based on single strategies.

Video recommendation with multi-gate mixture of experts soft actor critic

Existing research on cross-lingual retrieval cannot take good advantage of large-scale pretrained language models, such as multilingual BERT and XLM. The absence of cross-lingual passage-level relevance data for finetuning and the lack of query-document style pretraining are some of the key factors of this issue. Accordingly, embodiments of two novel retrieval-oriented pretraining tasks are presented herein to further pretrain cross-lingual language models for downstream retrieval tasks, such as cross-lingual ad-hoc retrieval (CUR) and cross-lingual question answering (CLQA). In one or more embodiments, distant supervision data was constructed from multilingual texts using section alignment to support retrieval-oriented language model pretraining. In one or more embodiments, directly finetuning language models on part of an evaluation collection was performed by making Transformers capable of accepting longer sequences. Experiments show that model embodiments significantly improve upon general multilingual language models in at least the cross-lingual retrieval setting and the cross-lingual transfer setting.

Growing community of inventors

Bellevue, WA, United States of America

Ping Li