Baidu USA LLC

Guang Chen

Weide Zhang

Tae Eun Choe

Ka Wai Tsoi

Yuliang Guo

Jun Zhu

I-Kuei Chen

A list of images is received. The images were captured by a sensor of an ADV chronologically while driving through a driving environment. A first image of the images is identified that includes a first object in a first dimension (e.g., larger size) detected by an object detector using an object detection algorithm. In response to the detection of the first object, the images in the list are traversed backwardly in time from the first image to identify a second image that includes a second object in a second dimension (e.g., smaller size) based on a moving trail of the ADV represented by the list of images. The second object is then labeled or annotated in the second image equivalent to the first object in the first image. The list of images having the labeled second image can be utilized for subsequent object detection during autonomous driving.

Method for automatically labeling objects in past frames based on object detection of a current frame for autonomous driving

During the autonomous driving, the movement trails or moving history of obstacles, as well as, an autonomous driving vehicle (ADV) may be maintained in a corresponding buffer. For the obstacles and the ADV, the vehicle states at different points in time are maintained and stored in one or more buffers. The vehicle states representing the moving trails or moving history of the obstacles and the ADV may be utilized to reconstruct a history trajectory of the obstacles and the ADV, which may be used for a variety of purposes. For example, the moving trails or history of obstacles may be utilized to determine lane configuration of one or more lanes of a road, particularly, in a rural area where the lane markings are unclear. The moving history of the obstacles may also be utilized predict the future movement of the obstacles, tailgate an obstacle, and infer a lane line.

Method for autonomously driving a vehicle based on moving trails of obstacles surrounding the vehicle

In some implementations, a method is provided. The method includes obtaining an image depicting an environment where an autonomous driving vehicle (ADV) may be located. The image comprises a plurality of line indicators. The plurality of line indicators represent one or more lanes in the environment. The image is part of training data for a neural network. The method also includes determining a plurality of line segments based on the plurality of line indicators. The method further includes determining a vanishing point within the image based on the plurality of line segments. The method further includes updating one or more of the image or metadata associated with the image to indicate a location of the vanishing point within the image.

Determining vanishing points based on lane lines

In some implementations, a method is provided. The method includes obtaining an image depicting an environment where an autonomous driving vehicle (ADV) is located. The method also includes determining, using a first neural network, a plurality of line indicators based on the image. The plurality of line indicators represent one or more lanes in the environment. The method further includes determining, using a second neural network, a vanishing point within the image based on the plurality of line segments. The second neural network is communicatively coupled to the first neural network. The plurality of line indicators is determined simultaneously with the vanishing point. The method further includes calibrating one or more sensors based of the autonomous driving vehicle based on the vanishing point.

Determining vanishing points based on feature maps

In one embodiment, a set of bounding box candidates are plotted onto a 2D space based on their respective dimension (e.g., widths and heights). The bounding box candidates are clustered on the 2D space based on the distribution density of the bounding box candidates. For each of the clusters of the bounding box candidates, an anchor box is determined to represent the corresponding cluster. A neural network model is trained based on the anchor boxes representing the clusters. The neural network model is utilized to detect or recognize objects based on images and/or point clouds captured by a sensor (e.g., camera, LIDAR, and/or RADAR) of an autonomous driving vehicle.

Method for determining anchor boxes for training neural network object detection models for autonomous driving

According to some embodiments, a system pre-processes, via a first thread, a captured image perceiving an environment surrounding the ADV obtained from an image capturing device of the ADV. The system processes, via a second thread, the pre-processed image with a corresponding depth image captured by a ranging device of the ADV using a machine learning model to detect vehicle lanes. The system post-processes, via a third thread, the detected vehicle lanes to track the vehicle lanes relative to the ADV. The system generates a trajectory based on a lane line of the tracked vehicle lanes to control the ADV autonomously according to the trajectory.

Lane/object detection and tracking perception system for autonomous vehicles

In one embodiment, in addition to detecting or recognizing an actual lane, a virtual lane is determined based on the current state or motion prediction of an ADV. A virtual lane may or may not be identical or similar to the actual lane. A virtual lane may represent the likely movement of the ADV in a next time period given the current speed and heading direction of the vehicle. If an object is detected that may cross a lane line of the virtual lane and is a closest object to the ADV, the object is considered as a CIPO, and an emergency operation may be activated. That is, even though an object may not be in the path of an actual lane, if the object is in the path of a virtual lane of an ADV, the object may be considered as a CIPO and subject to a special operation.

Method for detecting closest in-path object (CIPO) for autonomous driving

In response to a first image captured by a camera of an ADV, a horizon line is determined based on the camera's hardware settings, representing a vanishing point based on an initial or default pitch angle of the camera. One or more lane lines are determined based on the first image via a perception process performed on the first image. In response to a first input signal received from an input device, a position of the horizon line is updated based on the first input signal and a position of at least one of the lane lines is updated based on the updated horizon line. The input signal may represent an incremental adjustment for adjusting the position of the horizon line. A first calibration factor or first correction value is determined for calibrating a pitch angle of the camera based on a difference between the initial horizon line and the updated horizon line.

Sensor calibration system for autonomous driving vehicles

In one embodiment, a lane processing method and system identifies traffic lanes based on images. The images can be converted to lane markers where the lane markers are based on inner edges of the identified lanes. The lane markers can be used for steering, navigation, controlling and driving an automated driving vehicle (ADV). The markers can be associated with each other, in graphical space, to construct lane lines. Additional information, such as spatial and semantic information can be associated with each lane to further improve ADV planning and control.

Lane post-processing in an autonomous driving vehicle

In some implementations, a method is provided. The method includes obtaining an image depicting an environment where an autonomous driving vehicle (ADV) is located. The method also includes determining a vanishing point within the image using a neural network. The vanishing point is represented by the neural network as a relative distance to a center of the image. The method further includes calibrating one or more sensors of the autonomous driving vehicle based on the vanishing point.

Growing community of inventors

Sunnyvale, CA, United States of America

Guang Chen