Toyota Research Institute, Inc.

Massachusetts Institute of Technology

Toyota Jidosha Kabushiki Kaisha

Leland Stanford Junior University

The General Hospital Corporation

Toyota Jidosha Kabushiki

Guy Rosman

Stephen G McGill

Luke S Fletcher

John J Leonard

Daniela Rus

Simon A I Stent

Sertac Karaman

Igor Gilitschenski

Xin Huang

Paul Drews

Stephen G Mcgill

Alexander Amini

Deepak Edakkattil Gopinath

Brian C Williams

Adrien David Gaidon

Jonathan Decastro

Blake Warren Wulfe

Yusuke Sawamura

Noah J Epstein

John William Fisher, Iii

Thomas Balch

Simon Stent

Luke D Fletcher

Felix Maximilian Naser

Rares A Ambrus

William T Freeman

Tiffany L Chen

Gregory W Wornell

Nikos Arechiga Gonzalez

Daniel J Brooks

Shabnam Hakimi

Fredo Durand

Marco Pavone

Alyssa Pierson

Wolfram Burgard

John H Gideon

Moses Theodore Ort

Antonio Torralba

Emily Sarah Sumner

Boris Ivanovic

Minoru Brandon Araki

Ashkan Mohammadzadeh Jasour

Daniel Stonier

Jonathan A DeCastro

Xiao Li

Christina Liao

Thomas Ward

Katsuya Terahata

Ozanan R Meireles

Daniel A Hashimoto

Yutong Ban

Amine Elhafsi

Cristian-Ioan Vasile

Brandon Araki

Julian Straub

Oren Freifeld

John Gideon

Arjun Gupta

Brent Schlotfeldt

Andreas Buehler

A method for task-informed planning by a behavior planning system of a vehicle includes observing a previous trajectory of an agent within a distance from the vehicle. The method also includes predicting, by the behavior planning system, a set of potential trajectories for the agent and/or the vehicle based on observing the previous trajectory. The method further includes selecting, by the behavior planning system, a potential action from a set of potential actions associated with a task to be performed by the vehicle, each potential action being associated with a utility value based on the respective potential action and the set of potential trajectories, the selected potential action being associated with a highest utility value of respective utility values associated with the set of potential actions. The method still further includes controlling the vehicle to perform an action associated with the potential action selected by the behavior planning system.

Task-informed behavior planning

Systems and methods are provided for generating a statistical parameter representing a state of a surgical procedure from sensor data. Sensor data representing a time period. is received from a sensor. Numerical features representing the time period are generated from the sensor data. Each of a plurality of long short term memory units are updated according to the plurality of numerical features via a message passing process. The long short term memory units are connected to form a graph, with a first set of the long short term memory units representing a plurality of nodes of the graph and a second set of the long short term memory units representing a plurality of hyperedges of the graph. A statistical parameter representing a state of the surgical procedure for the time period is derived from an output of one of the long short term memory units and provided to a user.

Interpretation of intraoperative sensor data using concept graph neural networks

A method for indicating occlusion information at an ego agent includes observing a spatial area from a first viewpoint of one or more first sensors associated with the ego agent. The method also includes identifying the spatial area as an occluded area in accordance with observing the spatial area from a second viewpoint of the one or more first sensors after observing the spatial area from the first viewpoint. The method further includes receiving, from a target agent, a message indicating the spatial area is occluded from one or more second sensors associated with the target agent. The method still further includes transmitting, to the target agent in accordance with receiving the message, the occlusion information indicating information associated the spatial area based on identifying the spatial area as the occluded area.

Peer-to-peer occupancy estimation

Systems and methods of model or prediction algorithm selection are provided. An autonomous control system may include a perception component that, based on environmental inputs regarding an object(s), a vehicle's operating characteristics, etc., outputs a current state of the vehicle's surrounding environment. This in turn, is used as input to a prediction component comprising a plurality of prediction algorithms. The prediction component outputs a set of predictions regarding the trajectory of the object(s). Accordingly, for each object, a set of trajectories at specific timesteps may be generated by the different prediction algorithms which are input to a planner component. These trajectories may then be analyzed, compared, or otherwise processed to determine which trajectory regarding the object is most accurate. The prediction algorithm or model that produced the most accurate predicted trajectory may then be used for subsequent predictions/timesteps.

Systems and methods for multiple algorithm selection

A vehicle system includes one or more sensors configured to capture aspects of an environment and a computing device. The computing device is configured to receive information about the environment captured by the one or more sensors, determine one or more structures within the environment based on the received information, select a kernel that is parameterized for predicting a vehicle trajectory based on the one or more structures determined within the environment, and perform a convolution of the selected kernel and an array defining the environment, wherein the convolution predicts a future trajectory of a vehicle within the environment.

Convolution operator selection

System, methods, and other embodiments described herein relate to coordinating and optimizing prediction by a model using intermediate data and vehicle-to-vehicle (V2V) communications that improves bandwidth utilization. In one embodiment, a method includes processing acquired data, by a subject vehicle using a prediction model, to execute a vehicle task associated with navigating a driving scene. The method also includes extracting processed data according to the acquired data from intermediate activations of layers within the prediction model according to a cooperation model. The method also includes quantifying relevancy of the processed data, for a target vehicle, to select a subset using the cooperation model. The method also includes communicating the subset to a selected vehicle having the prediction model for additional prediction according to the relevancy and available resources to transmit the subset.

Systems and methods for optimizing coordination and communication resources between vehicles using models

Systems and methods for predicting a trajectory of a moving object are disclosed herein. One embodiment downloads, to a robot, a probabilistic hybrid discrete-continuous automaton (PHA) model learned as a deep neural network; uses the deep neural network to infer a sequence of high-level discrete modes and a set of associated low-level samples, wherein the high-level discrete modes correspond to candidate maneuvers for the moving object and the low-level samples are candidate trajectories; uses the sequence of high-level discrete modes and the set of associated low-level samples, via a learned proposal distribution in the deep neural network, to adaptively sample the sequence of high-level discrete modes to produce a reduced set of low-level samples; applies a sample selection technique to the reduced set of low-level samples to select a predicted trajectory for the moving object; and controls operation of the robot based, at least in part, on the predicted trajectory.

Systems and methods for predicting the trajectory of a moving object

Systems and methods are provided for an advanced driver-assistance system (ADAS) that obtains data from a plurality of sensors. In some embodiments, the system can retrieve data regarding a user's past interactions and analyze the data with the sensor data to determine the user's behavior. In some embodiments, the ADAS can determine whether a user is unaware of an ADAS feature based on this behavior and a prompt that recommends the ADAS feature. The user's response to this prompt may be incorporated into the user's behavior for future recommendations.

Vehicle-provided recommendations for use of ADAS systems

Systems and methods for learning and managing robot user interfaces are disclosed herein. One embodiment generates, based on input data including information about past interactions of a particular user with a robot and with existing HMIs of the robot, a latent space using one or more encoder neural networks, wherein the latent space is a reduced-dimensionality representation of learned behavior and characteristics of the particular user, and uses the latent space as input to train a decoder neural network associated with (1) a new HMI distinct from the existing HMIs or (2) a particular HMI among the existing HMIs to alter operation of the particular HMI. The trained first decoder neural network is deployed in the robot to control, at least in part, operation of the new HMI or the particular HMI in accordance with the learned behavior and characteristics of the particular user.

Systems and methods for learning and managing robot user interfaces

A mobile robot can be caused to move according to a planned trajectory. The mobile robot can be a vehicle. Information about agents in an environment of the mobile robot can be received from sensors. At a first time, a spatiotemporal graph can be produced. The spatiotemporal graph can represent relationships among the agents in the environment. The mobile robot can be one of the agents in the environment. Information from the spatiotemporal graph can be input to neural networks to produce information for a mixture of affine time-varying systems. The mixture of affine time-varying systems can represent an evolution of agent states of the agents. Using the mixture of affine time-varying systems and information associated with the first time, a prediction of the agent states at a second time can be calculated. The mobile robot can be caused to move according to the planned trajectory determined from the prediction.

Causing a mobile robot to move according to a planned trajectory determined from a prediction of agent states of agents in an environment of the mobile robot

A method of generating an output trajectory of an ego vehicle is described. The method includes extracting high-level features from a bird-view image of a traffic environment of the ego vehicle. The method also includes generating, using an automaton generative network, an automaton including an automaton state distribution describing a behavior of the ego vehicle in the traffic environment according to the high-level features. The method further includes generating the output trajectory of the ego vehicle according to extracted bird-view features of the bird-view image and the automaton state distribution describing the behavior of the ego vehicle in the traffic environment.

Method for learning an explainable trajectory generator using an automaton generative network

Systems and methods for generating efficient planning routes for vehicles, including autonomous and semi-autonomous vehicles are presented. A route planner may generate dynamic maps and routes that reduces the uncertainty of road-agent environmental and behavioral data in an efficient manner. Route planning may be accomplished using a statistical approach in which known data from one geographic or behavioral feature set may be used and relied upon by a vehicle in another geographical and behavioral context to estimate the environmental and behavioral data relevant to the vehicles current operation.

Dynamic map generation with focus on construction and localization field of technology

A system includes a camera configured to capture image data of an environment, a monitoring system configured to generate a gaze sequences of a subject, and a computing device communicatively coupled to the camera and the monitoring system. The computing device is configured to receive the image data from the camera and the gaze sequences from the monitoring system, implement a machine learning model comprising a convolutional encoder-decoder neural network configured to process the image data and a side-channel configured to inject the gaze sequences into a decoder stage of the convolutional encoder-decoder neural network, generate, with the machine learning model, a gaze probability density heat map, and generate, with the machine learning model, an attended awareness heat map.

Cognitive heat map: a model for driver situational awareness

A method includes receiving data relating to pedestrian activity at one or more locations outside of a crosswalk, analyzing the data, based on the data, identifying at least one location of the one or more locations as a constructive crosswalk, and controlling operation of an autonomous vehicle based on the at least one location of the constructive crosswalk.

Systems and methods for modeling pedestrian activity

A control system, computer-readable storage medium and method of preventing occlusion of and minimizing shadows on the driver's face for driver monitoring. The system includes a steering wheel, a plurality of fiberscopes arranged evenly spaced around the steering wheel, and one or more video cameras arranged at remote ends of the plurality of fiberscopes. Distal ends of the fiberscopes emerge to a surface of the steering wheel through holes that are perpendicular to an axis of rotation of the steering wheel. Each of the distal ends of the fiberscopes includes a lens. The system includes a plurality of light sources and an electronic control unit connected to the one or more video cameras and the light sources.

Multiple in-cabin cameras and lighting sources for driver monitoring

Growing community of inventors

Newton, MA, United States of America

Guy Rosman