Toyota Research Institute, Inc.

Toyota Jidosha Kabushiki Kaisha

Kla Tencor Corporation

Argo Ai Gmbh

Chao Fang

Kuan-Hui Lee

Kun-Hsin Chen

Charles Christopher Ochoa

Arjun Bhargava

Sudeep Pillai

Brian Duffy

Jia-En M Pan

Vitor Guizilini

Mark Davis Smith

Guy Hotson

Mohan Mahadevan

Amir Azordegan

Richard L Kwant

Daniele Molinari

Nicolas Cebron

Deva K Ramanan

Logan Michael Ellis

Amitoz Singh Dandiana

Constantin Franziskus Dominik Hubmann

T Wolfram Burgard

Saurabh Sharma

A method for vehicle prediction, planning, and control is described. The method includes separately encoding traffic state information at an intersection into corresponding traffic state latent spaces. The method also includes aggregating the corresponding traffic state latent spaces to form a generalized traffic geometry latent space. The method further includes interpreting the generalized traffic geometry latent space to form a traffic flow map including current and future vehicle trajectories. The method also includes decoding the generalized traffic geometry latent space to predict a vehicle behavior according to the traffic flow map based on the current and future vehicle trajectories.

Traffic signal understandings and representation for prediction, planning, and control

Systems, methods, and other embodiments described herein relate to a multi-task model that integrates recurrent models to improve handling of multi-sweep inputs. In one embodiment, a method includes acquiring sensor data from multiple modalities. The method includes separately encoding respective segments of the sensor data according to an associated one of the different modalities to form encoded features using separate encoders of a network. The method includes accumulating, in a detector, sparse features associated with sparse sensor inputs of the multiple modalities to densify the sparse features into dense features. The method includes providing observations according to the encoded features and the sparse features using the network.

Network for multisweep 3D detection

A method for controlling an ego vehicle in an environment includes associating, by a velocity model, one or more objects within the environment with a respective velocity instance label. The method also includes selectively, by a recurrent network of the taillight recognition system, focusing on a selected region of the sequence of images according to a spatial attention model for a vehicle taillight recognition task. The method further includes concatenating the selected region with the respective velocity instance label of each object of the one or more objects within the environment to generate a concatenated region label. The method still further planning a trajectory of the ego vehicle based on inferring, at a classifier of the taillight recognition system, an intent of each object of the one or more objects according to a respective taillight state of each object, as determined based on the concatenated region label.

Vehicle taillight recognition based on a velocity estimation

A method controlling an ego vehicle in an environment includes determining, via a flow model of a parked vehicle recognition system, a flow between a first representation of the environment and a second representation of the environment. The method also includes determining, via a velocity model of the parked vehicle recognition system, a velocity of a vehicle in the environment based on the flow. The method further includes determining, via a parked vehicle classification model of the parked vehicle recognition system, the vehicle is parked based on the velocity of the vehicle and one or more of features associated with the vehicle and/or the environment. The method still further includes planning a trajectory of the ego vehicle based on determining the vehicle is parked.

Parked car classification based on a velocity estimation

A method for generating a dense light detection and ranging (LiDAR) representation by a vision system includes receiving, at a sparse depth network, one or more sparse representations of an environment. The method also includes generating a depth estimate of the environment depicted in an image captured by an image capturing sensor. The method further includes generating, via the sparse depth network, one or more sparse depth estimates based on receiving the one or more sparse representations. The method also includes fusing the depth estimate and the one or more sparse depth estimates to generate a dense depth estimate. The method further includes generating the dense LiDAR representation based on the dense depth estimate and controlling an action of the vehicle based on identifying a three-dimensional object in the dense LiDAR representation.

Shared vision system backbone

Methods and systems for detecting objects near an autonomous vehicle (AV) are disclosed. An AV will capture an image. A trained network will process the image at a lower resolution and generate a first feature map that classifies object(s) within the image. The system will crop the image and use the network to process the cropped section at a higher resolution to generate a second feature map that classifies object(s) that appear within the cropped section. The system will crop the first feature map to match a corresponding region of the cropped section of the image. The system will fuse the cropped first and second feature maps to generate a third feature map. The system may output the object classifications in the third feature map to an AV system, such as a motion planning system that will use the object classifications to plan a trajectory for the AV.

Attentional sampling for long range detection in autonomous vehicles

System, methods, and other embodiments described herein relate to detection of traffic lights corresponding to a driving lane from views captured by multiple cameras. In one embodiment, a method includes estimating, by a first model using images from multiple cameras, positions and state confidences of traffic lights corresponding to a driving lane of a vehicle. The method also includes aggregating, by a second model, the state confidences and a multi-view stereo composition from geometric representations associated with the positions of the traffic lights. The method also includes assigning, by the second model according to the aggregating, a relevancy score computed for a candidate traffic light of the traffic lights to the driving lane. The method also includes executing a task by the vehicle according to the relevancy score.

Systems and methods for detecting traffic lights corresponding to a driving lane

System, methods, and other embodiments described herein relate to improving the detection of traffic lights associated with a driving lane using a camera instead of map data. In one embodiment, a method includes estimating, from an image using a first model, depth and orientation information of traffic lights relative to a driving lane of a vehicle. The method also includes computing, using a second model, relevancy scores for the traffic lights according to geometric inferences between the depth and the orientation information. The method also includes assigning, using the second model, a primary relevancy score for a light of the traffic lights associated with the driving lane according to the depth and the orientation information. The method also includes executing a control task by the vehicle according to the primary relevancy score and a state confidence, computed by the first model, for the light.

Systems and methods for detecting traffic lights of driving lanes using a camera and multiple models

System, methods, and other embodiments described herein relate to accurately distinguishing a traffic light from other illuminated objects in the traffic scene and detecting states using hierarchical modeling. In one embodiment, a method includes detecting, using a machine learning (ML) model, two-dimensional (2D) coordinates of illuminated objects identified from a monocular image of a traffic scene for control adaptation by a control model. The method also includes assigning, using the ML model, computed probabilities to the illuminated objects for categories within a hierarchical ontology of environmental lights associated with the traffic scene, wherein one of the probabilities indicates existence of a traffic light instead of a brake light in the traffic scene. The method also includes executing a task by the control model for a vehicle according to the 2D coordinates and the computed probabilities.

Systems and methods for detecting traffic lights using hierarchical modeling

A vehicle light classification system captures a sequence of images of a scene that includes a front/rear view of a vehicle with front/rear-side lights, determines semantic keypoints, in the images and associated with the front/rear-side lights, based on inputting the images into a first neural network, obtains multiple difference images that are each a difference between successive images from among the sequence of images, the successive images being aligned based on their respective semantic keypoints, and determines a classification of the front/rear-side lights based at least in part on the difference images by inputting the difference images into a second neural network.

Systems and methods for vehicle light signal classification

Described herein are systems and methods for determining if a vehicle is parked. In one example, a system includes a processor, a sensor system, and a memory. Both the sensor system and the memory are in communication with the processor. The memory includes a parking determination module having instructions that, when executed by the processor, cause the processor to determine, using a random forest model, when the vehicle is parked based on vehicle estimated features, vehicle learned features, and vehicle taillight features of the vehicle that are based on sensor data from the sensor system.

System and method for determining if a vehicle is parked

Methods and systems for generating a high resolution image for a specimen from a low resolution image of the specimen are provided. One system includes one or more computer subsystems configured for acquiring a low resolution image of a specimen. The system also includes one or more components executed by the one or more computer subsystems. The one or more components include a deep convolutional neural network that includes one or more first layers configured for generating a representation of the low resolution image. The deep convolutional neural network also includes one or more second layers configured for generating a high resolution image of the specimen from the representation of the low resolution image. The second layer(s) include a final layer configured to output the high resolution image and configured as a sub-pixel convolutional layer.

Generating high resolution images from low resolution images for semiconductor applications

A metrology system includes a controller communicatively coupled to a metrology tool. The controller may generate a three-dimensional model of a sample, generate a predicted metrology image corresponding to a predicted analysis of the sample with the metrology tool based on the three-dimensional model, evaluate two or more candidate metrology recipes for extracting the metrology measurement from the one or more predicted metrology images, select, based on one or more selection metrics, a metrology recipe from the two or more candidate metrology recipes for extracting a metrology measurement from an image of the structure from the metrology tool, receive an output metrology image of a fabricated structure from the metrology tool based on a metrology measurement of the fabricated structure, and extract the metrology measurement associated with the fabricated structure from the output metrology image based on the metrology recipe.

Growing community of inventors

Sunnyvale, CA, United States of America

Chao Fang