Honda Motor Co., Ltd.

Kumar Akash

Teruhisa Misu

Xingwei Wu

Zhaobo K Zheng

Sujitha Catherine Martin

Chun Ming Samson Ho

Yuning Qiu

Jundi Liu

Kaihong Wang

Haibei Zhu

Xiaofeng Gao

Manisha Natarajan

According to one aspect, systems, methods, and/or techniques associated with operator take-over prediction may include receiving a two-channel series of images of an operating environment through which a vehicle is travelling. A first series of images may be represented by labels corresponding to classified objects. A second series of images may be represented as a gaze heatmap or an eye tracking heatmap. Additionally, feeding encoded series of images through a three-dimensional (3D) convolutional neural network (CNN) to produce a first output, receiving sets of information corresponding to a first series of images and a second series of images in time, feeding sets of information through processing layers to produce additional outputs, concatenating the first output and the additional outputs to produce a concatenation output, and feeding the concatenation output through additional processing layers to generate an operator take-over prediction may be performed.

Operator take-over prediction

According to one aspect, systems and techniques for adaptive trust calibration may include usage of a driving style predictor, including a memory and a processor. The memory may store one or more instructions and the processor may execute one or more of the instructions stored on the memory to perform one or more acts, actions, or steps, such as receiving a current automated vehicle (AV) driving style, receiving an indication of an event and an associated event type, receiving an indication of a driver takeover, concatenating the current AV driving style and one or more of the event type or the driver takeover to generate an input, and passing the input through a neural network, which may include a gated recurrent unit (GRU), to generate a preference change associated with the AV driving style.

Adaptive trust calibration

According to one aspect, an adaptive driving style system may include a set of two or more sensors, a memory, and a processor. The set of two or more sensors may receive two or more sensor signals. The memory may store one or more instructions. The processor may execute one or more of the instructions stored on the memory to perform one or more acts, actions, or steps, including training a trust model using two or more of the sensor signals as input, training a preference model using the trust model and two or more of the sensor signals as input, and generating a driving style preference based on an adaptive driving style model including the trust model and the preference model.

Adaptive driving style

Aspects of adaptive trust calibration may include receiving a trust model for an occupant of an autonomous vehicle calculated based on occupant sensor data and a first scene context sensor data, and/or receiving a second scene context sensor data associated with an environment of the autonomous vehicle, determining an over trust scenario or an under trust scenario based on the trust model and a trust model threshold, and generating and implementing a human machine interface (HMI) action or a driving automation action based on the determination of the over trust scenario or the determination of the under trust scenario, and/or the second scene context sensor data.

A system and method for learning temporally consistent video synthesis using fake optical flow that include receiving data associated with a source video and a target video. The system and method also include processing image-to-image translation across domains of the source video and the target video and processing a synthesized temporally consistent video based on the image-to-image translation. The system and method further include training a neural network with data that is based on synthesizing of the source video and the target video.

System and method for learning temporally consistent video synthesis using fake optical flow

Systems and methods for clustering human trust dynamics are provided. In one embodiment, a computer implemented method for clustering human trust dynamics is provided. The computer implemented method includes receiving trust data for a plurality of participants interacting with one or more agents in an interaction. The computer implemented method also includes identifying a plurality of phases for the interaction. The computer implemented method further includes extracting features characterizing trust dynamics from the trust data for at least one interaction for each participant of the plurality participants. The at least one interaction is between the participant and an agent of the one or more agents. The computer implemented yet further includes assigning the features characterizing trust dynamics to a phase of the plurality of phases. The computer implemented method includes grouping a subset of the participant of the plurality of participants based on the on features characterizing trust dynamics.

Systems and methods for clustering human trust dynamics

A system and method for detecting a perceived level of driver discomfort in an automated vehicle that include receiving image data associated with a driving scene of an ego vehicle, dynamic data associated with an operation of the ego vehicle, and driver data associated with a driver of the ego vehicle during autonomous operation of the ego vehicle. The system and method also include analyzing the image data, the dynamic data, and the driver data and extracting features associated with a plurality of modalities. The system and method additionally include analyzing the extracted features and detecting the perceived level of driver discomfort. The system and method further include analyzing the perceived level of driver discomfort and detecting a probable driver takeover intent of the driver of the ego vehicle to takeover manual operation of the ego vehicle.

System and method for detecting a perceived level of driver discomfort in an automated vehicle

A system and method for providing an RNN-based human trust model that include receiving a plurality of inputs related to an autonomous operation of a vehicle and a driving scene of the vehicle and analyzing the plurality of inputs to determine automation variables and scene variables. The system and method also include outputting a short-term trust recurrent neural network state that captures an effect of the driver's experience with respect to an instantaneous vehicle maneuver and a long-term trust recurrent neural network state that captures the effect of the driver's experience with respect to the autonomous operation of the vehicle during a traffic scenario. The system and method further include predicting a take-over intent of the driver to take over control of the vehicle from an automated operation of the vehicle during the traffic scenario.

System and method for providing an RNN-based human trust model

An adaptive trust calibration based autonomous vehicle may include vehicle systems, a system behavior controller, and a driving automation controller. The system behavior controller may generate a driving automation signal indicative of a desired autonomous driving adaptation. The driving automation controller may control the vehicle systems based on parameters including a desired velocity, current velocity of the autonomous vehicle, desired minimum gap distance between the autonomous vehicle and a detected object, current gap distance gap between the autonomous vehicle and a detected object, relative velocity of the detected object with respect to the autonomous vehicle, desired time headway, desired maximum acceleration, desired braking deceleration, and an exponent. The driving automation controller may receive the driving automation signal and implement the desired autonomous driving adaptation via the vehicle systems by adjusting the parameters based on a type of object associated with the detected object.

A system and method for improving driver situation awareness prediction using human visual sensory and memory mechanism that includes receiving data associated with a driving scene of a vehicle and an eye gaze of a driver of the vehicle. The system and method also include analyzing the data and extracting features associated with objects located within the driving scene and determining a situational awareness score that is associated with a situational awareness of the driver with respect to each of the objects located within the driving scene. The system and method further include communicating control signals to electronically control at least one system of the vehicle based on the situational awareness score that is associated with each of the objects located within the driving scene.

System and method for improving driver situation awareness prediction using human visual sensory and memory mechanism

A system and method for determining object-wise situational awareness that includes receiving data associated with a driving scene of a vehicle, an eye gaze of a driver of the vehicle, and alerts that are provided to the driver of the vehicle. The system and method also includes analyzing the data and extracting features associated with dynamic objects located within the driving scene, the eye gaze of the driver of the vehicle, and the alerts provided to the driver of the vehicle. The system and method additionally includes determining a level of situational awareness of the driver with respect to the each of the dynamic objects based on the features. The system and method further includes communicating control signals to electronically control at least one component of the vehicle based on the situational awareness of the driver.

System and method for determining object-wise situational awareness

An autonomous driving agent is provided. The autonomous driving agent determines a set of observations from sensor information of a sensor system of a vehicle. The set of observations includes human attention information for a scene of surrounding environment and a level of human reliance as indicated by human inputs to the autonomous driving agent. The autonomous driving agent estimates, based on the set of observations, belief states for a first state of human trust on the autonomous driving agent and a second state of human's cognitive workload during journey. The autonomous driving agent selects, based on the estimated belief states, a first value for a first action associated with a level of automation transparency between a human user and the autonomous driving agent and controls a display system based on the selected first value to display a cue for calibration of the human trust on the autonomous driving agent.

Growing community of inventors

West Lafayette, IN, United States of America

Kumar Akash