Baidu USA LLC

Baidu.com Times Technology (beijing) Co. Ltd.

Peng Wang

Ruigang Yang

Xinjing Cheng

Wei Xu

Binbin Cao

Sibo Zhang

Miao Liao

Zhenheng Yang

Chenxu Luo

Yang Wang

Presented herein are novel embodiments for converting a given speech audio or text into a photo-realistic speaking video of a person with synchronized, realistic, and expressive body dynamics. In one or more embodiments, 3D skeleton movements are generated from the audio sequence using a recurrent neural network, and an output video is synthesized via a conditional generative adversarial network. To make movements realistic and expressive, the knowledge of an articulated 3D human skeleton and a learned dictionary of personal speech iconic gestures may be embedded into the generation process in both learning and testing pipelines. The former prevents the generation of unreasonable body distortion, while the later helps the model quickly learn meaningful body movement with a few videos. To produce photo-realistic and high-resolution video with motion details, a part-attention mechanism is inserted in the conditional GAN, where each detailed part is automatically zoomed in to have their own discriminators.

Personalized speech-to-video with three-dimensional (3D) skeleton regularization and expressive body poses

Presented are systems and methods for improving speed and quality of real-time per-pixel depth estimation of scene layouts from a single image by using an end-to-end Convolutional Spatial Propagation Network (CSPN). An efficient linear propagation model performs propagation using a recurrent convolutional operation. The affinity among neighboring pixels may be learned through a deep convolutional neural network (CNN). The CSPN may be applied to two depth estimation tasks, given a single image: (1) to refine the depth output of existing methods, and (2) to convert sparse depth samples to a dense depth map, e.g., by embedding the depth samples within the propagation procedure. The conversion ensures that the sparse input depth values are preserved in the final depth map and runs in real-time and is, thus, well suited for robotics and autonomous driving applications, where sparse but accurate depth measurements, e.g., from LiDAR, can be fused with image data.

Systems and methods for depth estimation via affinity learned with convolutional spatial propagation networks

Presented are deep learning-based systems and methods for fusing sensor data, such as camera images, motion sensors (GPS/IMU), and a 3D semantic map to achieve robustness, real-time performance, and accuracy of camera localization and scene parsing useful for applications such as robotic navigation and augment reality. In embodiments, a unified framework accomplishes this by jointly using camera poses and scene semantics in training and testing. To evaluate the presented methods and systems, embodiments use a novel dataset that is created from real scenes and comprises dense 3D semantically labeled point clouds, ground truth camera poses obtained from high-accuracy motion sensors, and pixel-level semantic labels of video camera images. As demonstrated by experimental results, the presented systems and methods are mutually beneficial for both camera poses and scene semantics.

Systems and methods for deep localization and segmentation with a 3D semantic map

Described herein are systems and methods for jointly learning geometry and motion with three-dimensional holistic understanding. In embodiments, such approaches enforce the inherent geometrical consistency during the learning process, yielding improved results for both tasks. In embodiments, three parallel networks are adopted to predict the camera motion (e.g., MotionNet), dense depth map (e.g., DepthNet), and per-pixel optical flow between consecutive frames (e.g., FlowNet), respectively. The information of 2D flow, camera pose, and depth maps, are fed into a holistic 3D motion parser (HMP) to disentangle and recover per-pixel 3D motion of both rigid background and moving objects. Various loss terms are formulated to jointly supervise the three networks. Embodiments of an efficient iterative training strategy are disclosed for better performance and more efficient convergence. Performance on depth estimation, optical flow estimation, odometry, moving object segmentation, and scene flow estimation demonstrates the effectiveness of the disclosed systems and methods.

Joint learning of geometry and motion with three-dimensional holistic understanding

Presented are systems and methods for improving speed and quality of real-time per-pixel depth estimation of scene layouts from a single image by using a 3D end-to-end Convolutional Spatial Propagation Network (CSPN). An efficient linear propagation model performs propagation using a recurrent convolutional operation. The affinity among neighboring pixels may be learned through a deep convolutional neural network (CNN). The CSPN may be applied to two depth estimation tasks, given a single image: (1) to refine the depth output of existing methods, and (2) to convert sparse depth samples to a dense depth map, e.g., by embedding the depth samples within the propagation procedure. For stereo depth estimation, the 3D CPSN is applied to stereo matching by adding a diffusion dimension over discrete disparity space and feature scale space. This aids the recovered stereo depth to generate more details and to avoid error matching from noisy appearance caused by sunlight, shadow, and similar effects.

Systems and methods for depth estimation using convolutional spatial propagation networks

Presented are systems and methods for 3D reconstruction from videos using an unsupervised learning framework for depth and normal estimation via edge-aware depth-normal consistency. In embodiments, this is accomplished by using a surface normal representation. Depths may be reconstructed in a single image by watching unlabeled videos. Depth-normal regularization constrains estimated depths to be compatible with predicted normals, thereby, yielding geometry-consistency and improving evaluation performance and training speed. In embodiments, a consistency term is solved by constructing depth-to-normal layer and normal-to-depth layers within a deep convolutional network (DCN). In embodiments, the depth-to-normal layer uses estimated depths to compute normal directions based on neighboring pixels. Given the estimated normals, the normal-to-depth layer may then output a regularized depth map. Both layers may be computed with awareness of edges within the image. Finally, to train the network, the photometric error and gradient smoothness for both depth and normal predictions may be applied.

Growing community of inventors

Sunnyvale, CA, United States of America

Peng Wang