Ford Global Technolgoies, LLC

Argo Ai Gmbh

Volkswagen Group of America Investments, LLC

Richard L Kwant

Guy Hotson

Siddharth Madan

Nicolas Cebron

Deva K Ramanan

Ravi Basavaraj

Neetu Agarwal

Brett Browning

Chao Fang

Hatem Alismail

Ersin Yumer

Dikpal Reddy

Hsin Miao

Shaojun Zhu

David James Chekan

Reza Azimi

Systems and methods for on-board selection of data logs for training a machine learning model. The methods include, by an autonomous vehicle, receiving sensor data logs corresponding to surroundings of the autonomous vehicle from a plurality of sensors, identifying one or more events within each sensor data log. The methods also include, for each sensor data log: analyzing features of the identified one or more events within that sensor data log for determining whether that sensor data log satisfies one or more usefulness criteria for training a machine learning model, and transmitting that sensor data log to a remote computing device for training the machine learning model if that sensor data log satisfies one or more usefulness criteria for training the machine learning model. The features can include spatial features, temporal features, bounding box inconsistencies, or map-based features.

Autonomous vehicle system for intelligent on-board selection of data for training a remote machine learning model

Methods and systems for determining whether a camera of an autonomous vehicle (AV) is calibrated are disclosed. The method includes using image frames captured by a camera of the AV and LIDAR point clouds captured by a LIDAR system of the AV to calculate an AV pose calibration metric. The method also includes: measuring a distance metric between a three-dimensional bounding box around an object and a two-dimensional bounding box in an image captured by the camera; using the AV pose calibration metric and the distance calibration metric to generate a confidence score; and in response to the confidence score being below a threshold, generating a signal indicating that the camera is not calibrated.

Systems and methods for validating camera calibration in real-time

Methods and systems for detecting objects near an autonomous vehicle (AV) are disclosed. An AV will capture an image. A trained network will process the image at a lower resolution and generate a first feature map that classifies object(s) within the image. The system will crop the image and use the network to process the cropped section at a higher resolution to generate a second feature map that classifies object(s) that appear within the cropped section. The system will crop the first feature map to match a corresponding region of the cropped section of the image. The system will fuse the cropped first and second feature maps to generate a third feature map. The system may output the object classifications in the third feature map to an AV system, such as a motion planning system that will use the object classifications to plan a trajectory for the AV.

Attentional sampling for long range detection in autonomous vehicles

Systems and methods are provided for operating a vehicle, is provided. The method includes, by a vehicle control system of the vehicle, identifying map data for a present location of the vehicle using a location of the vehicle and pose and trajectory data for the vehicle, identifying a field of view of a camera of the vehicle, and analyzing the map data to identify an object that is expected to be in the field of view of the camera. The method further includes, based on (a) a class of the object, (b) characteristics of a region of interest in the field of view of the vehicle, or (c) both, selecting an automatic exposure (AE) setting for the camera. The method additionally includes causing the camera to use the AE setting when capturing images of the object, and using the camera, capturing the images of the object.

Estimating auto exposure values of camera by prioritizing object of interest based on contextual inputs from 3D maps

This document discloses methods of training a classifier to identify traffic signal states in images captured be a vehicle. The vehicle can then use the identified states when making movement decisions when traveling in an environment. The system determines that a traffic signal is within a field of view of the camera (i.e., within an image). The system also receives a signal with signal phase and timing data for the traffic signal. The system processes the images to identify an image that includes the traffic signal. The system analyzes the signal data to determine a state of the traffic signal at the time of image capture, labels the image with a label of determined state, and passes the image and a label to a classifier in order to train the classifier.

Determining traffic light labels and classification quality from infrastructure signals

Systems and methods are provided for detecting a flashing light on one or more traffic signal devices. The method includes capturing a series of images of one or more traffic signal elements in a traffic signal device over a length of time. The method further includes, for each traffic signal element, analyzing the series of images to determine one or more time periods at which the traffic signal element is in an on state or an off state, and analyzing the time periods to determine one or more distinct on states and one or more distinct off states. The method further includes identifying one or more cycles correlating to a distinct on state immediately followed by a distinct off state, or a distinct off state immediately followed by a distinct on state, and, upon identifying a threshold number adjacent cycles, classifying the traffic signal element as a flashing light.

Light emitting diode flicker mitigation

This document describes methods to assign a collective state to a group of traffic signal devices that are concurrently detected by multiple cameras. The states may be color states or other states. The system will process the images to identify states of each of the devices in the images. When the devices exhibit more than one state, the system will determine an overall state for the group by generating a confidence score for each of the states. The system will select, from the multiple states, the state having a confidence score that exceeds a threshold. The system will then use the selected state to assign an overall state to the group of traffic signal devices. The system may use the overall state to generate a signal that will cause the vehicle to perform an action such as a motion control action, or output of an audible or visual alert.

Hierarchical processing of traffic signal face states

A method is disclosed for evaluating a classifier used to determine a traffic light signal state in images. The method includes, by a computer vision system of a vehicle, receiving at least one image of a traffic signal device of an imminent intersection. The traffic signal device includes a traffic signal face including one or more traffic signal elements. The method includes classifying, by a traffic light classifier (TLC), a classification state of the traffic signal face using labeled images correlated to the received at least one image. The classification state controls an operation of the vehicle at the intersection. The method includes evaluating a performance of the classifying of the classification state generated by the TLC. The evaluation is a label-free performance evaluation based on unlabeled images. The method includes training the TLC based on the evaluated performance.

Label-free performance evaluator for traffic light classifier system

Systems and methods are provided for identifying and representing a traffic signal device. The method includes determining a location and orientation of the vehicle and receiving a real world image. The method further includes analyzing information about the vehicle's location and environment and using this information and the vehicle's orientation to generate a raster image illustrating an approximation of a view of the real world image, including one or more traffic signal devices. Additionally, the method includes providing the real world image and the raster image as inputs to a neural network to classify a traffic signal device in the real world image as the primary traffic signal device and determine a set of coordinates indicating a location of the primary traffic signal device, generating a classified real world image which includes a bounding box indicating the set of coordinates, and receiving the classified real world image.

Using rasterization to identify traffic signal devices

Systems and methods are provided for representing a traffic signal device. The method includes receiving a digital image of a traffic signal device that includes one or more traffic signal elements, representing the traffic signal device as a raster image, each traffic signal element of the traffic signal device being represented by a mask corresponding to a location of the traffic signal element on the traffic signal device, representing each mask in a channel in the raster image, providing the raster image as an input to a neural network to classify a state for each of the one or more traffic signal elements, and receiving, from the neural network, a classified raster image, in which the classified raster image includes a plurality of masks, each mask representing a state of one of the one or more traffic signal elements.

Growing community of inventors

San Bruno, CA, United States of America

Richard L Kwant