Toyota Research Institute, Inc.

Massachusetts Institute of Technology

Toyota Jidosha Kabushiki Kaisha

Nationaluniversity of Ireland, Maynooth

John J Leonard

Luke S Fletcher

Guy Rosman

Stephen G McGill

Xin Huang

Brian C Williams

Simon A I Stent

Igor Gilitschenski

Gill A Pratt

Kazunori Nimura

Stephen G Mcgill

Yusuke Sawamura

Daniela Rus

Nikos Arechiga Gonzalez

Sertac Karaman

Jonathan Decastro

Alyssa Pierson

Moses Theodore Ort

John William Fisher, Iii

Simon Stent

Minoru Brandon Araki

Deepak Edakkattil Gopinath

Ashkan Mohammadzadeh Jasour

Jonathan A DeCastro

Michael Kaess

Daniel Stonier

Thomas John Whelan

John McDonald

Oren Freifeld

Julian Straub

Dehann Fourie

Katsuya Terahata

Sudeep Pillai

A method for task-informed planning by a behavior planning system of a vehicle includes observing a previous trajectory of an agent within a distance from the vehicle. The method also includes predicting, by the behavior planning system, a set of potential trajectories for the agent and/or the vehicle based on observing the previous trajectory. The method further includes selecting, by the behavior planning system, a potential action from a set of potential actions associated with a task to be performed by the vehicle, each potential action being associated with a utility value based on the respective potential action and the set of potential trajectories, the selected potential action being associated with a highest utility value of respective utility values associated with the set of potential actions. The method still further includes controlling the vehicle to perform an action associated with the potential action selected by the behavior planning system.

Task-informed behavior planning

Systems and methods for predicting a trajectory of a moving object are disclosed herein. One embodiment downloads, to a robot, a probabilistic hybrid discrete-continuous automaton (PHA) model learned as a deep neural network; uses the deep neural network to infer a sequence of high-level discrete modes and a set of associated low-level samples, wherein the high-level discrete modes correspond to candidate maneuvers for the moving object and the low-level samples are candidate trajectories; uses the sequence of high-level discrete modes and the set of associated low-level samples, via a learned proposal distribution in the deep neural network, to adaptively sample the sequence of high-level discrete modes to produce a reduced set of low-level samples; applies a sample selection technique to the reduced set of low-level samples to select a predicted trajectory for the moving object; and controls operation of the robot based, at least in part, on the predicted trajectory.

Systems and methods for predicting the trajectory of a moving object

A system includes a camera configured to capture image data of an environment, a monitoring system configured to generate a gaze sequences of a subject, and a computing device communicatively coupled to the camera and the monitoring system. The computing device is configured to receive the image data from the camera and the gaze sequences from the monitoring system, implement a machine learning model comprising a convolutional encoder-decoder neural network configured to process the image data and a side-channel configured to inject the gaze sequences into a decoder stage of the convolutional encoder-decoder neural network, generate, with the machine learning model, a gaze probability density heat map, and generate, with the machine learning model, an attended awareness heat map.

Cognitive heat map: a model for driver situational awareness

Systems and methods described herein relate to estimating risk associated with a vehicular maneuver. One embodiment acquires a geometric representation of an intersection including a lane in which a vehicle is traveling and at least one other lane; discretizes the at least one other lane into a plurality of segments; determines a trajectory along which the vehicle will travel; estimates a probability density function for whether a road agent external to the vehicle is present in the respective segments; estimates a traffic-conflict probability of a traffic conflict in the respective segments conditioned on whether an external road agent is present; estimates a risk associated with the vehicle following the trajectory by integrating a product of the probability density function and the traffic-conflict probability over the at least one other lane and the plurality of segments; and controls operation of the vehicle based, at least in part, on the estimated risk.

Systems and methods for estimating the risk associated with a vehicular maneuver

A system and method for generating a predicted vehicle trajectory includes a generative adversarial network configured to receive a trajectory vector of a target vehicle and generate a set of latent state vectors using the received trajectory vector and an artificial neural network. The latent state vectors each comprise a high-level sub-vector, Z. The GAN enforces Zto be correlated to an annotation coding representing semantic categories of vehicle trajectories. The GAN selects a subset, from the set of latent state vectors, using farthest point sampling and generates a predicted vehicle trajectory based on the selected subset of latent state vectors.

Methods and systems for diversity-aware vehicle motion prediction via latent semantic sampling

Systems and methods for parallel autonomy of a vehicle are disclosed herein. One embodiment receives input data, the input data including at least one of sensor data and structured input data; encodes the input data into an intermediate embedding space using a first neural network; inputs the intermediate embedding space to a first behavior model and a second behavior model, the first behavior model producing a first behavior output, the second behavior model producing a second behavior output; combines the first behavior output and the second behavior output using an ideal-behavior model to produce an ideal behavior for the vehicle; and controls one or more aspects of operation of the vehicle based, at least in part, on the ideal behavior.

Systems and methods for parallel autonomy of a vehicle

An embodiment takes the form of a vehicle that generates a data collection configuration for one or more vehicle sensors of a vehicle based on an estimated information gain to a neural network were the vehicle to provision the neural network with notional sensor data, and based on a vehicle resource consumption by the vehicle were the vehicle to provision the neural network with the notional sensor data. The notional sensor data comprises sensor data that would be collected from a given sensor among the vehicle sensors according to a respective sensor configuration of the given sensor. The vehicle collects sensor data from the vehicle sensors according to the generated data collection configuration.

Methods and systems for collecting sensor data according to a data collection configuration based on an estimated information gain

A vehicle system includes a steering wheel configured to output an output based on an input from a user, and a controller configured to: determine a target orientation of front wheels of a vehicle based on vehicle environment information, determine whether an orientation of the front wheels of the vehicle based on the output from the steering wheel deviates from the target orientation, disengage the steering wheel from the front wheels in response to determination that the orientation of the front wheels deviates from the target orientation, adjust the orientation of the front wheels to the target orientation and provide a feedback in response to adjusting the orientation of the front wheels to the target orientation.

Systems and methods for switching between a driver mode and an autonomous driving mode for a vehicle

Systems and methods described herein relate to controlling the operation of a vehicle. One embodiment generates predicted trajectories of the vehicle using first trajectory predictors based, at least in part, on first inputs; generates predicted trajectories of a road agent that is external to the vehicle using second trajectory predictors based, at least in part, on second inputs; integrates the predicted trajectories of the road agent into the first inputs to iteratively update the predicted trajectories of the vehicle and integrates the predicted trajectories of the vehicle into the second inputs to iteratively update the predicted trajectories of the road agent; and controls operation of the vehicle based, at least in part, on at least one of (1) the iteratively updated predicted trajectories of the vehicle and (2) the iteratively updated predicted trajectories of the road agent.

Systems and methods for controlling the operation of a vehicle

Systems and methods described herein relate to predicting a trajectory of a road agent external to a vehicle. One embodiment generates first and second predicted road-agent trajectories using respective first and second trajectory predictors based, at least in part, on a plurality of inputs including past road-agent trajectory information and vehicle sensor data; generates a confidence score for each predicted road-agent trajectory using a confidence estimator that includes a deep neural network, wherein generating the confidence scores includes computing them as a function of time within a predetermined temporal horizon; outputs the first and second predicted road-agent trajectories and their respective confidence scores; and controls operation of the vehicle based, at least in part, on one or more of the first predicted road-agent trajectory, the second predicted road-agent trajectory, the confidence score for the first predicted road-agent trajectory, and the confidence score for the second predicted road-agent trajectory.

Systems and methods for predicting the trajectory of a road agent external to a vehicle

A method of collecting data regarding the operation of a vehicle that includes receiving sensor data regarding one or more objects or events in a surrounding environment using one or more vehicle sensors, classifying, by one or more processors, the one or more detected objects or events, generating one or more vehicle inquiries based on the classification of the one or more detected objects or events, presenting one or more vehicle inquiries, and receiving user feedback to the one or more vehicle inquiries.

Systems and methods for incentivizing user-aided improvement of autonomous vehicle control systems and methods of operating a vehicle using the same

System, methods, and other embodiments described herein relate to modeling dynamic agents in a surrounding environment of an ego vehicle. In one embodiment, a method includes, in response to receiving sensor data including present observations of a road agent of the dynamic agents in the surrounding environment, identifying previous observations of the road agent from an electronic data store. The method includes estimating a future state of the road agent using at least the present observations and the previous observations of the road agent to compute the future state according to a probabilistic model comprised of a transition model that accounts for dynamic behaviors of the road agent to characterize transitions between states, and an agent model that accounts for actions of the road agent. The method includes controlling one or more vehicle systems of the ego vehicle according to the future state of the road agent.

Systems and methods for agent tracking

Vehicle systems and methods for assisting a driver making a turn at an intersection are disclosed. In one embodiment, a vehicle includes a plurality of sensors configured to output a plurality of operational signals, one or more processors, and one or more non-transitory memory modules communicatively coupled to the one or more processors. The processors store machine-readable instructions that, when executed, cause the one or more processors to receive a signal indicating a driver of the vehicle is attempting to make a turn at an intersection. The processors are further caused to access a behavioral profile that is representative of the intersection. The processors are also caused to develop a perception as well as a risk associated with the intersection based on the plurality of operational signals from the plurality of sensors and the behavioral profile of the intersection. The processors are further caused to calculate at least one maneuver.

Vehicle systems and methods for providing turn assistance at an intersection

A vehicle system for operating a vehicle is provided. The vehicle system includes a first output device configured to output a first output, and a second output device configured to output a second output. The vehicle system operates the vehicle based on the first output from the first output device while obtaining vehicle environment information, determines whether an autonomous take-over event occurs based on the vehicle environment information, operates the vehicle in an autonomous driving mode in response to determining that the autonomous takeover event occurred, determines whether the vehicle system receives a take-over signal from the second output device, and operates the vehicle based on the second output from the second output device in response to determining that the vehicle system received the take-over signal from the second output device.

A tunable and iterative stereo mapping technique is provided, capable of identifying disparities at or substantially faster than real-time (e.g., frame-rate of 120 Hz). The method includes identifying a plurality of points in an image, determining disparity values for each of the points in the image and generating a piece-wise planar mesh based on the points and their respective disparity values. A disparity interpolation can be performed on candidate planes using estimated plane parameters for the candidate planes and a disparity image can be generated having a plurality of regions based on the disparity interpolation. Multiple iterations can be performed until the image is reconstructed with an appropriate resolution based on predetermined thresholds. The thresholds can be modified to provide a tunable system by changing the threshold values to either increase a resolution of a final reconstructed image and/or increase a computation speed of the tunable and iterative stereo mapping technique.

Growing community of inventors

Cambridge, MA, United States of America

John J Leonard