Toyota Research Institute, Inc.

Toyota Jidosha Kabushiki Kaisha

Massachusetts Institute of Technology

Toyota Jidosha Kabushiki

Simon A I Stent

Guy Rosman

Luke S Fletcher

John H Gideon

Wojciech Matusik

Hiroshi Yasuda

Stephen G McGill

John J Leonard

Petr Kellnhofer

Antonio Torralba

Deepak Edakkattil Gopinath

Thomas Balch

Adrià Recasens

Jie Li

Tiffany L Chen

Daniel J Brooks

Jonathan Decastro

Shabnam Hakimi

Emily Sarah Sumner

Zhijian Liu

John Gideon

Katsuya Terahata

Systems and methods for driver dazzle detection for a subject vehicle, may include: using a plurality of sensors to gather data regarding driver characteristics and light source characteristics in an environment of the subject vehicle; evaluating sensor data from the plurality of sensors received to determine at least one of a driver characteristic and a light source characteristic; determining a level of dazzling of a driver of the vehicle based on the determined at least one of a driver characteristic and a light source characteristic; and engaging remedial action based on the determined level of dazzle of the driver of the vehicle, wherein the remedial action comprises at least one of switching a control of the vehicle from a manual drive mode to an autonomous drive mode and engaging an ADAS feature if it is detected that the determine level of dazzling is above a dazzling threshold.

Driver dazzle mitigation systems and methods

Extended reality content in a video can be varied based on driver attentiveness. The video can be of an external environment of a vehicle and can be presented in real-time on a display located within the vehicle. The display can be a video pass through display. The display can be an in-vehicle display, or it can be a part of a video pass-through extended reality headset. The video can present a view of an external environment of the vehicle as well as extended reality content. A level of attentiveness of a driver of the vehicle can be determined. An amount of the extended reality content presented in the video can be varied based on the level of attentiveness.

Varying extended reality content based on driver attentiveness

Systems and methods are provided for an advanced driver-assistance system (ADAS) that obtains data from a plurality of sensors. In some embodiments, the system can retrieve data regarding a user's past interactions and analyze the data with the sensor data to determine the user's behavior. In some embodiments, the ADAS can determine whether a user is unaware of an ADAS feature based on this behavior and a prompt that recommends the ADAS feature. The user's response to this prompt may be incorporated into the user's behavior for future recommendations.

Vehicle-provided recommendations for use of ADAS systems

Systems and methods for use of operator state conditions to generate, adapt or otherwise produce visual signals relaying information to the operator are disclosed. A monitoring system may observe and analyze a vehicle operator to determine an operator state. The monitoring system may transmit the observed driver state to an operator alert system that generates, conditions and controls the transmission of signals to the operator.

Systems and methods for driver state conditioned visual signals

Systems and methods for learning and managing robot user interfaces are disclosed herein. One embodiment generates, based on input data including information about past interactions of a particular user with a robot and with existing HMIs of the robot, a latent space using one or more encoder neural networks, wherein the latent space is a reduced-dimensionality representation of learned behavior and characteristics of the particular user, and uses the latent space as input to train a decoder neural network associated with (1) a new HMI distinct from the existing HMIs or (2) a particular HMI among the existing HMIs to alter operation of the particular HMI. The trained first decoder neural network is deployed in the robot to control, at least in part, operation of the new HMI or the particular HMI in accordance with the learned behavior and characteristics of the particular user.

Systems and methods for learning and managing robot user interfaces

Systems and methods for training remote photoplethysmography ('PPG') models that outputs a subject PPG signal based on a subject video clip of a subject are described herein. The system may have a processor and a memory in communication with the processor. The memory may include a training module having instructions that, when executed by the processor, cause the processor to train the remote PPG model in a self-supervised contrastive learning manner using an unlabeled video clip having a sequence of images of a face of a person.

System and method for training a remote photoplethysmography model

Extended reality content in a video can be varied based on a risk level of an external environment of a vehicle. The video can be of an external environment of a vehicle can be presented in real-time on a display located within the vehicle. The display can be a video pass through display. The display can be an in-vehicle display, or it can be a part of a video pass-through extended reality headset. The video can present a view of an external environment of the vehicle as well as extended reality content. A risk level of the external environment of the vehicle can be determined. An amount of the extended reality content presented in the video can be varied based on the risk level.

Varying extended reality content based on risk level of a driving environment

A system includes a camera configured to capture image data of an environment, a monitoring system configured to generate a gaze sequences of a subject, and a computing device communicatively coupled to the camera and the monitoring system. The computing device is configured to receive the image data from the camera and the gaze sequences from the monitoring system, implement a machine learning model comprising a convolutional encoder-decoder neural network configured to process the image data and a side-channel configured to inject the gaze sequences into a decoder stage of the convolutional encoder-decoder neural network, generate, with the machine learning model, a gaze probability density heat map, and generate, with the machine learning model, an attended awareness heat map.

Cognitive heat map: a model for driver situational awareness

A control system, computer-readable storage medium and method of preventing occlusion of and minimizing shadows on the driver's face for driver monitoring. The system includes a steering wheel, a plurality of fiberscopes arranged evenly spaced around the steering wheel, and one or more video cameras arranged at remote ends of the plurality of fiberscopes. Distal ends of the fiberscopes emerge to a surface of the steering wheel through holes that are perpendicular to an axis of rotation of the steering wheel. Each of the distal ends of the fiberscopes includes a lens. The system includes a plurality of light sources and an electronic control unit connected to the one or more video cameras and the light sources.

Multiple in-cabin cameras and lighting sources for driver monitoring

Embodiments described herein include systems and methods for predicting a transfer of control of a vehicle to a driver. A method includes receiving information about an environment of the vehicle, identifying at least one condition represented in the information about the environment of the vehicle that corresponds to at least one of one or more known conditions that lead to a handback of operational control of the vehicle to the driver, and predicting the transfer of control of the vehicle to the driver based on the at least one condition identified from the information about the environment of the vehicle.

Systems and methods for predicting control handback

Systems and methods for training a model are described herein. In one example, a system for training the model includes a processor and a memory in communication with the processor having a training module. The training module has instructions that cause the processor to determine a contrastive loss using a self-supervised contrastive loss function, adjust, based on the contrastive loss, model weights a visual backbone that generated feature maps and/or a textual backbone that generated feature vectors. The training module also has instructions that cause the processor to determine a localized loss using a supervised loss function that compares an image-caption attention map with visual identifiers and adjust, based on the localized loss, the model weights the visual backbone and/or the textual backbone.

System and method for training a model using localized textual supervision

A system for predicting a hazardous event from road-scene data includes an electronic control unit configured to implement a neural network and a camera communicatively coupled to the electronic control unit, wherein the camera generates the road-scene data. The electronic control unit is configured to receive the road-scene data from the camera, and predict, with the neural network, an occurrence of the hazardous event within the road-scene data from the camera.

Systems and methods for simulating instantaneous attentive human responses to road hazards for improved early warning systems and fleet data harvesting

Systems and methods for an intelligent camera system are provided. A method includes receiving, from a first camera in a vehicle, view data corresponding to an area from a vantage point of the vehicle. The method further includes detecting a region of interest from the view data provided by the first camera. The method also includes providing the region of interest to a second camera in the vehicle. The method further includes receiving, from the second camera, zoom view data corresponding to a zoom view of the region of interest.

Methods and systems providing an intelligent camera system

Systems, vehicles and methods for determining wrong direction driving are disclosed. In one embodiment, a system for determining a vehicle traveling in a wrong direction includes one or more sensors that produce sensor data, one or more processors, and one or more non-transitory computer-readable medium storing computer readable-instructions. When the computer-readable instructions are executed by the one or more processors, the computer-readable instructions cause the one or more processors to determine one or more lanes within a roadway using the sensor data, determine a direction of travel of the one or more lanes using the sensor data, and identify a non-compliant vehicle traveling in a direction in the one or more lanes that is different from the determined direction of travel in the one or more lanes.

Growing community of inventors

Cambridge, MA, United States of America

Simon A I Stent