Baidu USA LLC

Hongyi Sun

Jinghao Miao

Jiangtao Hu

Kecheng Xu

Jiacheng Pan

Qi Luo

Yifei Jiang

Jiaming Tao

Zejun Lin

Wesley Reynolds

Liangliang Zhang

Dong Li

Jinyun Zhou

Jiaxuan Xu

Shu Jiang

Yajia Zhang

Wei Wei Wang

Wei Wang

Yu Long Cao

Feng Quan Liu

Weiman Lin

Xiangquan Xiao

Yu Wang

Li Zhuang

Youling Zou

Yu Cao

Feng Liu

In one embodiment, an emergency vehicle detection system can be provided in the ADV travelling on a road to detect the presence of an emergency vehicle in a surrounding environment of the ADV using both audio data and visual data. The emergency vehicle detection system can use a trained neutral network to independently generate a detection result from the audio data, and use another trained network to independently generate another detection result from the visual data. The emergency vehicle detection system can fuse the two detection results to determine the position and moving direction of the emergency vehicle. The ADV can take appropriate actions in response to the position and moving direction of the emergency vehicle.

Emergency vehicle audio and visual detection post fusion

According to various embodiments, systems, methods, and media for evaluating an open space planner in an autonomous vehicle are disclosed. In one embodiment, an exemplary method includes receiving, at a profiling application, a record file recorded by the ADV while driving in an open space using the open space planner, and a configuration file specifying parameters of the ADV; extracting planning messages and prediction messages from the record file, each extracted message being associated with the open space planner. The method further includes generating features from the planning message and the prediction messages in view of the specified parameters of the ADV; and calculating statistical metrics from the features. The statistical metrics are then provided to an automatic tuning framework for tuning the open space planner.

Open space planner profiling tool for autonomous vehicle

Sound signals are received by one or more microphones disposed at an ADV. The sound signals are analyzed to extract a feature of a road on which the ADV is driving. A road condition of the road is determined based on the extracted feature. A path planning and speed planning is performed to generate a trajectory based on the road condition. The ADV is controlled to drive autonomously according to the generated trajectory.

Audio-based technique to sense and detect the road condition for autonomous driving vehicles

In one embodiment, a process is performed during controlling Autonomous Driving Vehicle (ADV). A confidence level associated with a sensed obstacle is determined. If the confidence level is below a confidence threshold, and a distance between the ADV and a potential point of contact with the sensed obstacle is below a distance threshold, then performance of a driving decision is delayed. Otherwise, the driving decision is performed to reduce risk of contact with the sensed obstacle.

Delay decision making for autonomous driving vehicles in response to obstacles based on confidence level and distance

Sound signals that are not audible are generated by one or more speakers disposed in the vehicle. Reflected sound signals from a driver or a passenger of the vehicle that are not audible are received by one or more microphones disposed in the vehicle. A behavior-induced acoustic pattern is detected based on the reflected ultrasound signals. The behavior-induced acoustic pattern is analyzed to recognize a behavior of the driver or the passenger of the vehicle. A response or an alert is generated according to the recognized behavior of the driver or the passenger in the vehicle.

In-vehicle acoustic monitoring system for driver and passenger

An obstacle state evolution of a spatial position of a moving obstacle over a period of time is determined. A lane-obstacle relation evolution of the moving obstacle with each of one or more lanes near the moving obstacle over the period of time is further determined. An intended movement of the moving obstacle is predicted based on the obstacle state evolution and the lane-obstacle evolution. Thereafter, a trajectory of the ADV is planned to control the ADV to avoid a collision with the moving obstacle based on the predicted intended movement of the moving obstacle. The above process is iteratively performed for each of the moving obstacles detected within a predetermined proximity of the ADV.

Lane-attention: predicting vehicles' moving trajectories by learning their attention over lanes

Systems and methods for generating labelled audio data and onboard validation of the labelled audio data utilizing an autonomous driving vehicle (ADV) while the ADV is operating within a driving environment are disclosed. The method includes recording a sound emitted by an object within the driving environment of the ADV, and converting the recorded sound into audio samples. The method further includes labelling the audio samples, and refining the labelled audio samples to produce refined labelled audio data. The refined labelled audio data is utilized to subsequently train a machine learning algorithm to recognize a sound source during autonomous driving of the ADV. The method further includes generating a performance profile of the refined labelled audio data based at least on the audio samples, a position of the object, and a relative direction of the object. The position of the object and the relative direction of the object are determined by a perception system of the ADV.

Audio logging for model training and onboard validation utilizing autonomous driving vehicle

In one embodiment, an autonomous driving system of an ADV perceives a driving environment surrounding the ADV based on sensor data obtained from various sensors, including detecting one or more lanes and at least a moving obstacle or moving object. For each of the lanes identified, an NN lane feature encoder is applied to the lane information of the lane to extract a set of lane features. For a given moving obstacle, an NN obstacle feature encoder is applied to the obstacle information of the obstacle to extract a set of obstacle features. Thereafter, a lane selection predictive model is applied to the lane features of each lane and the obstacle features of the moving obstacle to predict which of the lanes the moving obstacle intends to select.

Neural network with lane aggregation for lane selection prediction of moving objects during autonomous driving

According to various embodiments, systems, methods, and mediums for operating an autonomous driving vehicles (ADV) are described. The embodiments use a number of machine learning models to extract features individually from audio data and visual data captured by sensors mounted on the ADV, and then to fuse these extracted features to create a concatenated feature vectors. The concatenated feature vector is provided to a multiplayer perceptron (MLP) as input to generate a detection result related to the presence of an emergency vehicle in the surrounding environment. The detection result can be used by the ADV to take appropriate actions to comply with the local traffic rules.

Machine learning model to fuse emergency vehicle audio and visual detection

According to various embodiments, systems and methods described in the disclosure combine mapped features with point cloud features to improve object detection precision of an autonomous driving vehicle (ADV). The map features and the point cloud features can be extracted from a perception area of the ADV within a particular angle view at each driving cycle based on a position of the ADV. The map features and the point cloud features can be concatenated and provided to a neutral network for object detections.

Comprehensive and efficient method to incorporate map features for object detection with LiDAR

Systems and methods for sound source detection and localization utilizing an autonomous driving vehicle (ADV) are disclosed. The method includes receiving audio data from a number of audio sensors mounted on the ADV. The audio data comprises sounds captured by the audio sensors and emitted by one or more sound sources. Based on the received audio data, the method further includes determining a number of sound source information. Each sound source information comprises a confidence score associated with an existence of a specific sound. The method further includes generating a data representation to report whether there exists the specific sound within the driving environment of the ADV. The data representation comprises the determined sound source information. The received audio data and the generated data representation are utilized to subsequently train a machine learning algorithm to recognize the specific sound source during autonomous driving of the ADV in real-time.

Sound source detection and localization for autonomous driving vehicle

A moving obstacle such as a vehicle within a proximity of an intersection and one or more exits of the intersection are identified. An obstacle state evolution of a spatial position of the moving obstacle over a period of time is determined. For each of the exits, an intersection exit encoding of the exit is determined based on intersection exit features of the exit. An aggregated exit encoding based on aggregating all of the intersection exit encodings for the exits is determined. For each of the exits, an exit probability of the exit that the moving obstacle likely exits the intersection through the exit is determined based on the obstacle state evolution and the aggregated exit encoding. Thereafter, a trajectory of the ADV is planned to control the ADV to avoid a collision with the moving obstacle based on the exit probabilities of the exits.

Method for predicting exiting intersection of moving obstacles for autonomous driving vehicles

In response to perceiving a moving object, one or more possible object paths of the moving object are determined based on the prior movement predictions of the moving object, for example, using a machine-learning model, which may be created based on a large amount of driving statistics of different vehicles. For each of the possible object paths, a set of trajectory candidates is generated based on a set of predetermined accelerations. Each of the trajectory candidates corresponds to one of the predetermined accelerations. A trajectory cost is calculated for each of the trajectory candidates using a predetermined cost function. One of the trajectory candidates having the lowest trajectory cost amongst the trajectory candidates is selected. An ADV path is planned to navigate the ADV to avoid collision with the moving object based on the lowest costs of the possible object paths of the moving object.

Method for predicting movement of moving objects relative to an autonomous driving vehicle

In one embodiment, a set of predetermined driving parameters is determined from a set of driving statistics data collected from a number of vehicles, which may be driven by human drivers. For each pair of the predetermined driving parameters, a distribution of the pair of driving parameters is plotted based on their relationship on a two-dimensional (2D) distribution space. The 2D distribution space is partitioned into a number of grid cells, each grid cell representing a particular pair of driving parameters. For each of the grid cells, a probability is calculated that the pair of driving parameter likely falls in the grid cell. A grid table is generated corresponding to the pair of driving parameters. The grid table can be utilized during the autonomous driving at real-time or during simulation to determine a ride stability of an autonomous driving vehicle (ADV) in view of the pair of driving parameters.

Method for determining ride stability of an autonomous driving system controlling an autonomous driving vehicle

A moving object such as a vehicle is identified within an intersection having multiple exits. The moving object and the intersection and its exits may be identified based on sensor data obtained from various sensors mounted on an ADV. An exit coordinate map is generated based on the orientation of the moving object and a relative position of each of the exits of the intersection with respect to the current position of the moving object. For each of the exits, an exit probability of the exit that the moving object likely exits the intersection using the exit coordinate map. Thereafter, a trajectory of the ADV is planned to navigate through the intersection to avoid the collision with the moving object based on the exit probabilities of the exits of the intersection. The above process is iteratively performed for each of the moving objects detected within the proximity of the intersection.

Growing community of inventors

Sunnyvale, CA, United States of America

Hongyi Sun