Baidu USA LLC

Baidu.com Times Technology (beijing) Co. Ltd.

Wei Xu

Jiang Wang

Zhiheng Huang

Yi Yang

Haonan Yu

Junhua Mao

Haichao Zhang

Peng Wang

Lei Wang

Binbin Cao

Jie Zhou

Ruigang Yang

Zihang Dai

Haoyuan Gao

Kan Chen

Zhenheng Yang

Lei Li

Presented herein are embodiments of a multimodal Recurrent Neural Network (m-RNN) model for generating novel image captions. In embodiments, it directly models the probability distribution of generating a word given a previous word or words and an image, and image captions are generated according to this distribution. In embodiments, the model comprises two sub-networks: a deep recurrent neural network for sentences and a deep convolutional network for images. In embodiments, these two sub-networks interact with each other in a multimodal layer to form the whole m-RNN model. The effectiveness of an embodiment of model was validated on four benchmark datasets, and it outperformed the state-of-the-art methods. In embodiments, the m-RNN model may also be applied to retrieval tasks for retrieving images or captions.

Intelligent image captioning

Described herein are systems and methods for grounded natural language learning in an interactive setting. In embodiments, during a learning process, an agent learns natural language by interacting with a teacher and learning from feedback, thus learning and improving language skills while taking part in the conversation. In embodiments, a model is used to incorporate both imitation and reinforcement by leveraging jointly sentence and reward feedback from the teacher. Various experiments are conducted to validate the effectiveness of a model embodiment.

Listen, interact, and talk: learning to speak via interaction

Described herein are systems and methods for interactive language acquisition with one-shot concept learning through a conversational environment. Supervised language learning is limited by the ability of capturing mainly the statistics of training data, and is hardly adaptive to new scenarios or flexible for acquiring new knowledge without inefficient retraining or catastrophic forgetting. In one or more embodiments, conversational interaction serves as a natural interface both for language learning and for novel knowledge acquisition. Embodiments of a joint imitation and reinforcement approach are disclosed for grounded language learning through interactive conversation. An agent trained with this approach is able to actively acquire information by asking questions about novel objects and use the just-learned knowledge in subsequent conversations in a one-shot fashion. Results compared with other methods verified the effectiveness of embodiments disclosed herein.

Systems and methods for interactive language acquisition with one-shot visual concept learning through a conversational game

Presented are deep learning-based systems and methods for fusing sensor data, such as camera images, motion sensors (GPS/IMU), and a 3D semantic map to achieve robustness, real-time performance, and accuracy of camera localization and scene parsing useful for applications such as robotic navigation and augment reality. In embodiments, a unified framework accomplishes this by jointly using camera poses and scene semantics in training and testing. To evaluate the presented methods and systems, embodiments use a novel dataset that is created from real scenes and comprises dense 3D semantically labeled point clouds, ground truth camera poses obtained from high-accuracy motion sensors, and pixel-level semantic labels of video camera images. As demonstrated by experimental results, the presented systems and methods are mutually beneficial for both camera poses and scene semantics.

Systems and methods for deep localization and segmentation with a 3D semantic map

Embodiments of a multimodal question answering (mQA) system are presented to answer a question about the content of an image. In embodiments, the model comprises four components: a Long Short-Term Memory (LSTM) component to extract the question representation; a Convolutional Neural Network (CNN) component to extract the visual representation; an LSTM component for storing the linguistic context in an answer, and a fusing component to combine the information from the first three components and generate the answer. A Freestyle Multilingual Image Question Answering (FM-IQA) dataset was constructed to train and evaluate embodiments of the mQA model. The quality of the generated answers of the mQA model on this dataset is evaluated by human judges through a Turing Test.

Multilingual image question answering

Presented are systems and methods for 3D reconstruction from videos using an unsupervised learning framework for depth and normal estimation via edge-aware depth-normal consistency. In embodiments, this is accomplished by using a surface normal representation. Depths may be reconstructed in a single image by watching unlabeled videos. Depth-normal regularization constrains estimated depths to be compatible with predicted normals, thereby, yielding geometry-consistency and improving evaluation performance and training speed. In embodiments, a consistency term is solved by constructing depth-to-normal layer and normal-to-depth layers within a deep convolutional network (DCN). In embodiments, the depth-to-normal layer uses estimated depths to compute normal directions based on neighboring pixels. Given the estimated normals, the normal-to-depth layer may then output a regularized depth map. Both layers may be computed with awareness of edges within the image. Finally, to train the network, the photometric error and gradient smoothness for both depth and normal predictions may be applied.

Systems and methods for unsupervised learning of geometry from images using depth-normal consistency

Described herein are systems and methods for determining how to automatically answer questions like 'Where did Harry Potter go to school?' Carefully built knowledge graphs provide rich sources of facts. However, it still remains a challenge to answer factual questions in natural language due to the tremendous variety of ways a question can be raised. Presented herein are embodiments of systems and methods for human inspired simple question answering (HISQA), a deep-neural-network-based methodology for automatic question answering using a knowledge graph. Inspired by human's natural actions in this task, embodiments first find the correct entity via entity linking, and then seek a proper relation to answer the question&#x2014;both achieved by deep gated recurrent networks and neural embedding mechanism.

Systems and methods for human inspired simple question answering (HISQA)

Described herein are systems and methods that address the task of learning novel visual concepts, and their interactions with other concepts, from a few images with sentence descriptions. Using linguistic context and visual features, embodiments are able to efficiently hypothesize the semantic meaning of new words and add them to model word dictionaries so that they can be used to describe images which contain these novel concepts. In the experiments, it was shown that the tested embodiments effectively learned novel visual concepts from a few examples without disturbing the previously learned concepts.

Systems and methods for fast novel visual concept learning from sentence descriptions of images

Described herein are systems and methods that exploit hierarchical Recurrent Neural Networks (RNNs) to tackle the video captioning problem; that is, generating one or multiple sentences to describe a realistic video. Embodiments of the hierarchical framework comprise a sentence generator and a paragraph generator. In embodiments, the sentence generator produces one simple short sentence that describes a specific short video interval. In embodiments, it exploits both temporal- and spatial-attention mechanisms to selectively focus on visual elements during generation. In embodiments, the paragraph generator captures the inter-sentence dependency by taking as input the sentential embedding produced by the sentence generator, combining it with the paragraph history, and outputting the new initial state for the sentence generator.

Systems and methods for video paragraph captioning using hierarchical recurrent neural networks

Described herein are systems and methods for human-like language acquisition in a compositional framework to implement object recognition or navigation tasks. Embodiments include a method for a model to learn the input language in a grounded and compositional manner, such that after training the model is able to correctly execute zero-shot commands, which have either combination of words in the command never appeared before, and/or new object concepts learned from another task but never learned from navigation settings. In embodiments, a framework is trained end-to-end to learn simultaneously the visual representations of the environment, the syntax and semantics of the language, and outputs actions via an action module. In embodiments, the zero-shot learning capability of a framework results from its compositionality and modularity with parameter tying.

Deep compositional frameworks for human-like language acquisition in virtual environments

Described herein are systems and methods for generating and using attention-based deep learning architectures for visual question answering task (VQA) to automatically generate answers for image-related (still or video images) questions. To generate the correct answers, it is important for a model's attention to focus on the relevant regions of an image according to the question because different questions may ask about the attributes of different image regions. In embodiments, such question-guided attention is learned with a configurable convolutional neural network (ABC-CNN). Embodiments of the ABC-CNN models determine the attention maps by convolving image feature map with the configurable convolutional kernels determined by the questions semantics. In embodiments, the question-guided attention maps focus on the question-related regions and filters out noise in the unrelated regions.

Growing community of inventors

Saratoga, CA, United States of America

Wei Xu