Baidu USA LLC

Weide Zhang

Guang Chen

Tae Eun Choe

Ka Wai Tsoi

Yuliang Guo

Jun Zhu

Yu Huang

Hsien-Ting Cheng

Jing Zhai

I-Kuei Chen

Xiao Bo

Jun Zhou

Tony Han

Richard Chun-Ching Wang

Zhenhua Yu

A list of images is received. The images were captured by a sensor of an ADV chronologically while driving through a driving environment. A first image of the images is identified that includes a first object in a first dimension (e.g., larger size) detected by an object detector using an object detection algorithm. In response to the detection of the first object, the images in the list are traversed backwardly in time from the first image to identify a second image that includes a second object in a second dimension (e.g., smaller size) based on a moving trail of the ADV represented by the list of images. The second object is then labeled or annotated in the second image equivalent to the first object in the first image. The list of images having the labeled second image can be utilized for subsequent object detection during autonomous driving.

Method for automatically labeling objects in past frames based on object detection of a current frame for autonomous driving

During the autonomous driving, the movement trails or moving history of obstacles, as well as, an autonomous driving vehicle (ADV) may be maintained in a corresponding buffer. For the obstacles and the ADV, the vehicle states at different points in time are maintained and stored in one or more buffers. The vehicle states representing the moving trails or moving history of the obstacles and the ADV may be utilized to reconstruct a history trajectory of the obstacles and the ADV, which may be used for a variety of purposes. For example, the moving trails or history of obstacles may be utilized to determine lane configuration of one or more lanes of a road, particularly, in a rural area where the lane markings are unclear. The moving history of the obstacles may also be utilized predict the future movement of the obstacles, tailgate an obstacle, and infer a lane line.

Method for autonomously driving a vehicle based on moving trails of obstacles surrounding the vehicle

In one embodiment, a system to accelerate batch-normalized convolutional neural network (CNN) models is disclosed. The system extracts a plurality of first groups of layers from a first CNN model, each group of the first groups having a first convolutional layer and a first batch-norm layer. For each group of the plurality of first groups, the system calculates a first scale vector and a first shift vector based on the first batch-norm layer, and generates a second convolutional layer representing the corresponding group of the plurality of first groups based on the first convolutional layer and the first scale and the first shift vectors. The system generates an accelerated CNN model based on the second convolutional layer corresponding to the plurality of the first groups, such that the accelerated CNN model is utilized subsequently to classify an object perceived by an autonomous driving vehicle (ADV).

Offline combination of convolutional/deconvolutional and batch-norm layers of convolutional neural network models for autonomous driving vehicles

In some implementations, a method is provided. The method includes obtaining an image depicting an environment where an autonomous driving vehicle (ADV) may be located. The image comprises a plurality of line indicators. The plurality of line indicators represent one or more lanes in the environment. The image is part of training data for a neural network. The method also includes determining a plurality of line segments based on the plurality of line indicators. The method further includes determining a vanishing point within the image based on the plurality of line segments. The method further includes updating one or more of the image or metadata associated with the image to indicate a location of the vanishing point within the image.

Determining vanishing points based on lane lines

In some implementations, a method is provided. The method includes obtaining an image depicting an environment where an autonomous driving vehicle (ADV) is located. The method also includes determining, using a first neural network, a plurality of line indicators based on the image. The plurality of line indicators represent one or more lanes in the environment. The method further includes determining, using a second neural network, a vanishing point within the image based on the plurality of line segments. The second neural network is communicatively coupled to the first neural network. The plurality of line indicators is determined simultaneously with the vanishing point. The method further includes calibrating one or more sensors based of the autonomous driving vehicle based on the vanishing point.

Determining vanishing points based on feature maps

In one embodiment, a set of bounding box candidates are plotted onto a 2D space based on their respective dimension (e.g., widths and heights). The bounding box candidates are clustered on the 2D space based on the distribution density of the bounding box candidates. For each of the clusters of the bounding box candidates, an anchor box is determined to represent the corresponding cluster. A neural network model is trained based on the anchor boxes representing the clusters. The neural network model is utilized to detect or recognize objects based on images and/or point clouds captured by a sensor (e.g., camera, LIDAR, and/or RADAR) of an autonomous driving vehicle.

Method for determining anchor boxes for training neural network object detection models for autonomous driving

According to some embodiments, a system pre-processes, via a first thread, a captured image perceiving an environment surrounding the ADV obtained from an image capturing device of the ADV. The system processes, via a second thread, the pre-processed image with a corresponding depth image captured by a ranging device of the ADV using a machine learning model to detect vehicle lanes. The system post-processes, via a third thread, the detected vehicle lanes to track the vehicle lanes relative to the ADV. The system generates a trajectory based on a lane line of the tracked vehicle lanes to control the ADV autonomously according to the trajectory.

Lane/object detection and tracking perception system for autonomous vehicles

In one embodiment, in addition to detecting or recognizing an actual lane, a virtual lane is determined based on the current state or motion prediction of an ADV. A virtual lane may or may not be identical or similar to the actual lane. A virtual lane may represent the likely movement of the ADV in a next time period given the current speed and heading direction of the vehicle. If an object is detected that may cross a lane line of the virtual lane and is a closest object to the ADV, the object is considered as a CIPO, and an emergency operation may be activated. That is, even though an object may not be in the path of an actual lane, if the object is in the path of a virtual lane of an ADV, the object may be considered as a CIPO and subject to a special operation.

Method for detecting closest in-path object (CIPO) for autonomous driving

In response to a first image captured by a camera of an ADV, a horizon line is determined based on the camera's hardware settings, representing a vanishing point based on an initial or default pitch angle of the camera. One or more lane lines are determined based on the first image via a perception process performed on the first image. In response to a first input signal received from an input device, a position of the horizon line is updated based on the first input signal and a position of at least one of the lane lines is updated based on the updated horizon line. The input signal may represent an incremental adjustment for adjusting the position of the horizon line. A first calibration factor or first correction value is determined for calibrating a pitch angle of the camera based on a difference between the initial horizon line and the updated horizon line.

Sensor calibration system for autonomous driving vehicles

In one embodiment, a lane processing method and system identifies traffic lanes based on images. The images can be converted to lane markers where the lane markers are based on inner edges of the identified lanes. The lane markers can be used for steering, navigation, controlling and driving an automated driving vehicle (ADV). The markers can be associated with each other, in graphical space, to construct lane lines. Additional information, such as spatial and semantic information can be associated with each lane to further improve ADV planning and control.

Lane post-processing in an autonomous driving vehicle

In some implementations, a method is provided. The method includes obtaining an image depicting an environment where an autonomous driving vehicle (ADV) is located. The method also includes determining a vanishing point within the image using a neural network. The vanishing point is represented by the neural network as a relative distance to a center of the image. The method further includes calibrating one or more sensors of the autonomous driving vehicle based on the vanishing point.

Representing vanishing points using relative distances

Described is a system that matches a query with a set of sentences. When a query is received, the system may extract features from the query including one or more words as tokens and retrieve a set of candidate sentences as potential results for the query. For example, the query may be in the form of a question, and the candidate sentences may be alternative phrasings of the query that potentially match the user's search intent. The relevant set of candidate sentences may be determined based on using multiple relevancy scores and the system may rank the candidates according to an overall or aggregate score. Accordingly, the set of results to a query may be provided by recognizing the sentence as a whole and, for example when the query is a question, the interrogative intent of the query.

Matching a query to a set of sentences using a multidimensional relevancy determination

In one embodiment, a method or system generates a high resolution 3-D point cloud to operate an autonomous driving vehicle (ADV) from a low resolution 3-D point cloud and camera-captured image(s). The system receives a first image captured by a camera for a driving environment. The system receives a second image representing a first depth map of a first point cloud corresponding to the driving environment. The system determines a second depth map by applying a convolutional neural network model to the first image. The system generates a third depth map by applying a conditional random fields model to the first image, the second image and the second depth map, the third depth map having a higher resolution than the first depth map such that the third depth map represents a second point cloud perceiving the driving environment surrounding the ADV.

High resolution 3D point clouds generation based on CNN and CRF models

In one embodiment, a method or system generates a high resolution 3-D point cloud to operate an autonomous driving vehicle (ADV) from a low resolution 3-D point cloud and camera-captured image(s). The system receives a first image captured by a camera for a driving environment. The system receives a second image representing a first depth map of a first point cloud corresponding to the driving environment. The system downsamples the second image by a predetermined scale factor until a resolution of the second image reaches a predetermined threshold. The system generates a second depth map by applying a convolutional neural network (CNN) model to the first image and the downsampled second image, the second depth map having a higher resolution than the first depth map such that the second depth map represents a second point cloud perceiving the driving environment surrounding the ADV.

High resolution 3D point clouds generation from downsampled low resolution LIDAR 3D point clouds and camera images

In one embodiment, a method or system generates a high resolution 3-D point cloud to operate an autonomous driving vehicle (ADV) from a low resolution 3-D point cloud and camera-captured image(s). The system receives a first image captured by a camera for a driving environment. The system receives a second image representing a first depth map of a first point cloud corresponding to the driving environment. The system upsamples the second image by a predetermined scale factor to match an image scale of the first image. The system generates a second depth map by applying a convolutional neural network (CNN) model to the first image and the upsampled second image, the second depth map having a higher resolution than the first depth map such that the second depth map represents a second point cloud perceiving the driving environment surrounding the ADV.

Growing community of inventors

Sunnyvale, CA, United States of America

Weide Zhang