Baidu USA LLC

Beijing Baidu Netcom Science and Technology Co., Ltd.

Liangjun Zhang

Jinxin Zhao

Liyang Wang

Jin Fang

Xibin Song

Dingfu Zhou

Sibo Zhang

Zhixian Ye

Miao Liao

Jiahong Yuan

Qingkai Lu

Ruitao Song

Qiangqiang Guo

Autonomous excavator has developed rapidly in recent years because of a shortage of labor and hazardous working environments for operating excavators. Presented herein are embodiments of a novel hierarchical planning system for autonomous machines, such as excavators. In one or more embodiments, the overall planning system comprises a high-level task planner for task division and base movement planning, and general sub-task planners with motion primitives, which include both arm and base movement in the case of an excavator. Using embodiments of the system architecture, experiments were performed for the trench and pile removal tasks in the real world and for the large-scale material loading tasks in a simulation environment. The results show that the system architecture embodiments and planner method embodiments generate effective task and motion plans that perform well in autonomous excavation.

Hierarchical planning for autonomous machinery on various tasks

Presented herein are embodiments of a vision-based object perception system for activity analysis, safety monitoring, or both. Embodiments of the perception subsystem detect multi-class objects (e.g., construction machines and humans) in real-time while estimating the poses and actions of the detected objects. Safety monitoring embodiments and object activity analysis embodiments may be based on the perception result. To evaluate the performance of embodiments, a dataset was collected including multi-class of objects in different lighting conditions with human annotations. Experimental results show that the proposed action recognition approach outperforms the state-of-the-art approaches on top-1 accuracy by about 5.18%.

Vision-based safety monitoring and/or activity analysis

A method of determining a vehicle travel trajectory, an electronic device, a storage medium and a vehicle, which relate to a field of an artificial intelligence technology, in particular to a field of autonomous driving and intelligent transportation. A specific implementation solution includes: determining an initial path information for a vehicle; optimizing the initial path information to generate a target optimized path information; determining an optimized mapping relationship for velocity according to the target optimized path information and a first energy consumption constraint parameter; and determining an optimized trajectory for the vehicle according to the target optimized path information and the optimized mapping relationship for velocity.

Method of determining vehicle travel trajectory, electronic device, storage medium, and vehicle

Presented herein are embodiments of a two-stage methodology that integrates data-driven imitation learning and model-based trajectory optimization to generate optimal trajectories for autonomous excavators. In one or more embodiments, a deep neural network using demonstration data to mimic the operation patterns of human experts under various terrain states, including their geometry shape and material type. A stochastic trajectory optimization methodology is used to improve the trajectory generated by the neural network to ensure kinematics feasibility, improve smoothness, satisfy hard constraints, and achieve desired excavation volumes. Embodiments were tested on a Franka robot arm equipped with a bucket end-effector. Embodiments were also evaluated on different material types, such as sand and rigid blocks. Experimental results showed that embodiments of the two-stage methodology that comprises combining expert knowledge and model optimization increased the excavation weights by up to 24.77% with low variance.

Imitation learning and model integrated trajectory planning

Embodiments of a learning-based excavation planning method are disclosed for excavating rigid objects in clutter, which is challenging due to high variance of geometric and physical properties of objects, and large resistive force during the excavation. A convolutional neural network is utilized to predict a probability of excavation success. Embodiments of a sampling-based optimization method are disclosed for planning high-quality excavation trajectories by leveraging the learned prediction model. To reduce simulation-to-real gap for excavation learning, voxel-based representations of an excavation scene are used. Excavation experiments were performed in both simulation and real world to evaluate the learning-based excavation planners. Experimental results show that embodiments of the disclosed method may plan high-quality excavations for rigid objects in clutter and outperform baseline methods by large margins.

Excavation learning for rigid objects in clutter

The present application provides a model training method and apparatus, and a prediction method and apparatus, and it relates to fields of artificial intelligence, deep learning, image processing, and autonomous driving. The model training method includes: inputting a first sample image of sample images into a depth information prediction model, and acquiring depth information of the first sample image; acquiring inter-image posture information based on a second sample image of the sample images and the first sample image; acquiring a projection image corresponding to the first sample image, at least according to the inter-image posture information and the depth information; and acquiring a loss function by determining a function for calculating a similarity between the second sample image and the projection image, and training the depth information prediction model using the loss function.

Model training method and apparatus, and prediction method and apparatus

A radar point cloud data processing method and device, an apparatus, and storage medium are provided, which are related to technical fields of radar point cloud, automatic driving, and deep learning. An implementation includes: determining a target location area where a target object is located by utilizing a target detection box in the radar point cloud data; removing each point of the target object in the target location area from the radar point cloud data; and adding an object model to the target location area. By applying embodiments of the present disclosure, richer radar point cloud data may be obtained by removing the target object from the radar point cloud data and adding the needed three-dimensional model to the target location area in the radar point cloud data.

Radar point cloud data processing method and device, apparatus, and storage medium

The present disclosure discloses an engineering machinery equipment, and a method, system, and storage medium for safety control thereof, and relates to the field of artificial intelligence, automatic control, and engineering machinery technologies. A method can include: acquiring spatial sensing data of a work area; performing obstacle detection based on the spatial sensing data to determine a position of an obstacle within the work area; determining a safe working range of a mechanical structural component of the engineering machinery equipment based on the position of the obstacle within the work area; and controlling a working range of the mechanical structural component of the engineering machinery equipment based on the safe working range.

Engineering machinery equipment, and method, system, and storage medium for safety control thereof

A scalable solution to robot behavioral navigation following natural language instructions is presented. An example of the solution includes: receiving, by a pre-trained sequential prediction model, a navigation graph of the task environment, instructions in natural language and an initial location of the robot in the navigation graph, wherein the navigation graph comprises nodes indicating locations in the task environment, coordinates of the nodes, and edges indicating connectivity between the locations; and predicting sequentially, by the pre-trained sequential prediction model, a sequence of single-step behaviors executable by the robot to navigate the robot from the initial location to a destination.

Natural language based indoor autonomous navigation

The present disclosure discloses an engineering machinery equipment, and a method, system, and storage medium for operation trajectory planning thereof, and relates to the field of artificial intelligence, automatic control, and engineering machinery technologies. A method can include: acquiring three-dimensional sensing data of a material pile, to construct a three-dimensional model of the material pile based on the three-dimensional sensing data; determining a loading operation position of the engineering machinery equipment on the material pile based on the three-dimensional model of the material pile and structural design information of the engineering machinery equipment; and acquiring position information of a mechanical structural component of the engineering machinery equipment, and performing operation trajectory planning based on the position information of the mechanical structural component and the loading operation position, to generate an operation trajectory of the mechanical structural component executing a material loading operation.

Engineering machinery equipment, and method, system, and storage medium for operation trajectory planning thereof

Presented herein are novel approaches to synthesize video of the speech from text. In a training phase, embodiments build a phoneme-pose dictionary and train a generative neural network model using a generative adversarial network (GAN) to generate video from interpolated phoneme poses. In deployment, the trained generative neural network in conjunction with the phoneme-pose dictionary convert an input text into a video of a person speaking the words of the input text. Compared to audio-driven video generation approaches, the embodiments herein have a number of advantages: 1) they only need a fraction of the training data used by an audio-driven approach; 2) they are more flexible and not subject to vulnerability due to speaker variation; and 3) they significantly reduce the preprocessing, training, and inference times.

Text-driven video synthesis with phonetic dictionary

Described herein are systems and methods for inverse reinforcement learning to leverage the benefits of model-based optimization method and model-free learning method. Embodiments of a framework combining human behavior model with model predictive control are presented. The framework takes advantage of feature identification capability of a neural network to determine the reward function of model predictive control. Furthermore, embodiments of the present approach are implemented to solve the practical autonomous driving longitudinal control problem with simultaneous preference on safe execution and passenger comfort.

Inverse reinforcement learning with model predictive control

Embodiments of the present disclosure disclose a method, apparatus, device, and storage medium for controlling a guide robot, and relate to the field of artificial intelligence, robots, and multi-sensor fusion technologies. A specific embodiment of the method includes: acquiring a state of the guide robot, a state of a user, and a position of an obstacle; generating a state update equation for a combined system of the guide robot and the user based on the state of the guide robot and the state of the user; generating a collision-free global path based on the position of the obstacle; generating a control command based on the state update equation for the combined system and the collision-free global path; and driving the guide robot to move based on the control command.

Method, apparatus, device, and storage medium for controlling guide robot

Described herein are embodiments of a neural network-based task planner (TaskNet) for autonomous vehicle. Given a high-level task, the TaskNet planner decomposes it into a sequence of sub-tasks, each of which is further decomposed into task primitives with specifications. TaskNet comprises a first model for predicating the global sequence of working area to cover large terrain, and a second model for determining local operation order and specifications for each operation. The neural models may include convolutional layers for extracting features from grid map-based environment representation, and fully connected layers to combine extracted features with past sequences and predict the next sub-task or task primitive. Embodiments of the TaskNet are trained using an excavation trace generator and evaluate its performance using a 3D physically-based terrain and excavator simulator. Experiment results show TaskNet may effectively learn common task decomposition strategies and generate suitable sequences of sub-tasks and task primitives.

Growing community of inventors

Sunnyvale, CA, United States of America

Liangjun Zhang