Toyota Research Institute, Inc.

Simon Stent

Guy Rosman

Stephen G McGill

Luke S Fletcher

Wojciech Matusik

John J Leonard

Petr Kellnhofer

Antonio Torralba

Deepak Edakkattil Gopinath

An embodiment takes the form of a vehicle that generates a data collection configuration for one or more vehicle sensors of a vehicle based on an estimated information gain to a neural network were the vehicle to provision the neural network with notional sensor data, and based on a vehicle resource consumption by the vehicle were the vehicle to provision the neural network with the notional sensor data. The notional sensor data comprises sensor data that would be collected from a given sensor among the vehicle sensors according to a respective sensor configuration of the given sensor. The vehicle collects sensor data from the vehicle sensors according to the generated data collection configuration.

Methods and systems for collecting sensor data according to a data collection configuration based on an estimated information gain

An embodiment takes the form of a training server that presents a video comprising a plurality of frames, each comprising a respective scene representation of a scene at a respective time. The scene representations comprise respective representations of a feature in the scene. The training server presents a respective gaze representation of a driver gaze for each frame. The gaze representations comprise respective representations of driver gaze locations at the times of the respective scene representations. The training server generates an awareness prediction via a neural network based on the driver gaze locations, the awareness prediction reflecting a predicted driver awareness of the feature. The training server receives an awareness indication associated with the video and the gaze representations, and trains the neural network based on a comparison of the awareness prediction with the awareness indication.

Methods and system for predicting driver awareness of a feature in a scene

A system for determining the gaze direction of a subject includes a camera, a computing device and a machine-readable instruction set. The camera is positioned in an environment to capture image data of head of a subject. The computing device is communicatively coupled to the camera and the computing device includes a processor and a non-transitory computer-readable memory. The machine-readable instruction set is stored in the non-transitory computer-readable memory and causes the computing device to: receive image data from the camera, analyze the image data using a convolutional neural network trained on an image dataset comprising images of a head of a subject captured from viewpoints distributed around up to 360-degrees of head yaw, and predict a gaze direction vector of the subject based upon a combination of head appearance and eye appearance image data from the image dataset.

Systems and methods for gaze tracking from arbitrary viewpoints

Growing community of inventors

Cambridge, MA, United States of America

Simon Stent