Toyota Research Institute, Inc.

Toyota Jidosha Kabushiki Kaisha

Deepak Edakkattil Gopinath

Guy Rosman

Luke S Fletcher

Simon A I Stent

Stephen G McGill

Tiffany L Chen

John J Leonard

Daniel J Brooks

Jonathan Decastro

Shabnam Hakimi

Emily Sarah Sumner

Simon Stent

Katsuya Terahata

Systems and methods for learning and managing robot user interfaces are disclosed herein. One embodiment generates, based on input data including information about past interactions of a particular user with a robot and with existing HMIs of the robot, a latent space using one or more encoder neural networks, wherein the latent space is a reduced-dimensionality representation of learned behavior and characteristics of the particular user, and uses the latent space as input to train a decoder neural network associated with (1) a new HMI distinct from the existing HMIs or (2) a particular HMI among the existing HMIs to alter operation of the particular HMI. The trained first decoder neural network is deployed in the robot to control, at least in part, operation of the new HMI or the particular HMI in accordance with the learned behavior and characteristics of the particular user.

Systems and methods for learning and managing robot user interfaces

A system includes a camera configured to capture image data of an environment, a monitoring system configured to generate a gaze sequences of a subject, and a computing device communicatively coupled to the camera and the monitoring system. The computing device is configured to receive the image data from the camera and the gaze sequences from the monitoring system, implement a machine learning model comprising a convolutional encoder-decoder neural network configured to process the image data and a side-channel configured to inject the gaze sequences into a decoder stage of the convolutional encoder-decoder neural network, generate, with the machine learning model, a gaze probability density heat map, and generate, with the machine learning model, an attended awareness heat map.

Cognitive heat map: a model for driver situational awareness

An embodiment takes the form of a training server that presents a video comprising a plurality of frames, each comprising a respective scene representation of a scene at a respective time. The scene representations comprise respective representations of a feature in the scene. The training server presents a respective gaze representation of a driver gaze for each frame. The gaze representations comprise respective representations of driver gaze locations at the times of the respective scene representations. The training server generates an awareness prediction via a neural network based on the driver gaze locations, the awareness prediction reflecting a predicted driver awareness of the feature. The training server receives an awareness indication associated with the video and the gaze representations, and trains the neural network based on a comparison of the awareness prediction with the awareness indication.

Methods and system for predicting driver awareness of a feature in a scene

Growing community of inventors

Chicago, IL, United States of America

Deepak Edakkattil Gopinath