Rockwell Collins, Inc.

Smart Information Flow Technologies, LLC

Raytheon Company

Peggy Wu

Christopher L George

Michael P Matessa

Timothy J Wittkop

Wade T Johnson

Arjun Harsha Rao

Jeffrey Mathew Rye

Harry Bromberg Funk

Christopher A Miller

Tammy Elizabeth Ott

Sonja Maria Schmer-Galunder

Ahmad F Osman

Sarah Barber

Felix B Turcios

Bryan C Schultz

Ganesh Sundaramoorthi

Brigid A Blakeslee

Gianna Avdic McIntire

Andrew J Radlbeck

Brett W Israelsen

Andrew Radlbeck

Katharine G Woodruff

A method includes obtaining multiple data items from multiple heterogeneous data sources including at least one sensor, where the multiple data items are related to operation of an autonomy. The method also includes processing the multiple data items using a machine learning model to determine a likelihood that an operator associated with the autonomy will override an action of the autonomy during the operation of the autonomy. The method further includes outputting the likelihood for review by a user.

System and method for predicting manned unmanned teaming (MUM-T) autonomy overrides

An operator safety system (OSS) monitors the physiological well-being of an aircraft pilot or vehicle operator, and has engaged and disengaged operational states (the engaged state associated with active monitoring). The OSS includes a smart engage/disengage system incorporating presence sensors (e.g., visual, seat-based) installed in the cockpit or control area of the vehicle. When the presence sensors determine that the pilot/operator is no longer present in their seat but the OSS is still engaged, the OSS prompts the operator to disengage the OSS (e.g., via single-touch display interface or remotely to a mobile device). When the presence sensors determine that the operator is present or seated, but the OSS is disengaged, the OSS prompts the operator to re-engage the OSS.

Vehicle operator monitoring system with smart active engagement and disengagement

A system is described. The system may detect whether a user within a flight deck is authorized to control the aircraft. The user may receive differing levels of authority to control the aircraft depending on whether the user is authorized or unauthorized. The system may also control the aircraft depending on the various phases of the aircraft, including in flight and on the ground. The system may also be used to unlock the controls for the aircraft. The system may unlock the control using biometric information or nonbiometric information. The system may include a camera which is added to a flight deck. The camera may capture images of users within the flight deck. Faces within the images may be detected and compared against authorized users for locking or unlocking the aircraft controls.

System and method to detect authorized user

A computer system records eye tracking data and identifies movements in the eye tracking data to determine gaze and pupil dynamics. Eye tracking data is correlated with a current task and predetermined vigilance requirements. The system determines if the user is exhibiting an appropriate level of vigilance based on the task or is becoming fixated. When fixation is detected, the system may engage in remedial action. A task flow diagram represents the operator tasks. Interactions between the user and the instrumentation are used to estimate the probability distribution of the task the user is currently conducting. The system correlates eye tracking data and physiological data such as electroencephalogram (EEG) and functional near-infrared spectroscopy (fNIRs) to determine neuroactivity. Monitoring neuroactivity reduces the probability of a false positive for fixation.

Pupil dynamics, physiology, and context for estimating vigilance

An aircraft-based pilot safety system (PSS) includes cameras or other gaze sensors fixed at an aircraft pilot and configured to capture an image stream focused on the pilot's eyes. The gaze sensors continually assess the gaze direction of the pilot (e.g., toward a display, instrument panel, and/or indicator within the cockpit that the pilot is currently looking at or focusing on) and thereby can establish when the pilot is executing a scan pattern and if that scan pattern is nominal for the current flight context by comparing the scan pattern to context-specific reference scan patterns. If, for example, the pilot's gaze deviates from where it should be (e.g., as determined by the current flight segment) or the current scan pattern is interrupted, the system may prompt the pilot to redirect their gaze or resume the scan pattern.

Pilot safety system with context-sensitive scan pattern monitoring and alerting

A computer-implemented method is described. The method is be implemented by processors of an aircraft system. The method includes receiving images of an eye and a lighting configuration associated with a cockpit of an aircraft. The method further includes detecting a position of the eye within each of the images. The method further includes compensating for a pupillary light response of the eye based on the position of the eye within the image and the lighting configuration. By compensating for the pupillary light response, a fatigue level of the operator is estimated with reduced noise.

Cockpit display ambient lighting information for improving gaze estimation

A system and method for continuous real-time assessment of the situational awareness of an aircraft operator incorporates gaze sensors to determine the current gaze target (or sequence of gaze targets) of the operator, e.g., which interfaces the operator is looking at. The system receives operational context from aircraft systems indicative of current events and conditions both internal and external to the aircraft (e.g., operational status, mission or flight plan objectives, weather conditions). Based on the determined gaze targets and coterminous operational context, the system evaluates the situational awareness of the operator relative to the operational context, e.g., perceptive of the operational context; comprehending the operational context and its implications, and projecting the operator's perception and comprehension into responsive action and second-order ramifications according to task models indicative of expected behavior.

System and method for assessing operator situational awareness via context-aware gaze detection

A computerized system for recognizing the need for cognitive tests due to a physiological event and administering such tests includes a plurality of biometric monitoring devices and a pilot input device. When a physiological event is identified, the system selects an appropriate battery of tests and automatically administers those tests. The system offers the opportunity to automate pilot evaluation for single pilot operations, and to reduce the burden on a co-pilot. The cognitive tests may be specific to the physiological event, and may be compared to a pilot-specific profile when evaluating the results.

Cognitive battery for return to service

A system may include a vehicle. The vehicle may include an array of haptic devices. The system may further include at least one processor configured to: determine a location of an object or occurrence relative to the user; based at least on the location of the object or occurrence relative to the user, select at least one haptic device of the array of haptic devices to be driven and function as a directional haptic alert to the user, wherein the directional haptic alert is indicative of a direction from the user toward the object or occurrence; and output at least one command to cause a driving of the selected at least one haptic device, wherein the driving of the selected at least one haptic device is perceivable by the user as the directional haptic alert.

Vehicular directional alerting system and method using haptic alerts and optional multi-modal alerts

A system and method for inferring operator intent by detecting operator focus incorporates cameras positioned within a cockpit or control space of a vehicle and oriented at an operator of the vehicle. The cameras capture images of the operator in a control seat; the images are analyzed (either individually or sequentially) to determine a gaze and/or body pose of the operator (including, e.g., a position and orientation of the torso and limbs). By comparing the determined gaze and/or body pose to the positions and orientations of potential focus targets within the control space (e.g., windows, display units, and/or control panels that the operator may engage with visually and/or physically), the system predicts the most likely future focus target or targets: what the operator is most likely to visually and/or physically engage with next. Operator intent may be further analyzed to identify potentially abnormal or anomalous behavior.

System and method for gaze and pose detection to anticipate operator intent

A context driven alerting system and method is described in accordance with one or more embodiments of the present disclosure. The alerting system and method may consider a pilot's physiological, psychological, and behavioral state during a given mission context. The system may include biometric information which is fused with a context of the flight. The context may be based on one or more of a mission profile data or an aircraft state data. The fused data may be time synchronized and provided to an alerting algorithm. The alerting algorithm may then provide an alert to the pilot which includes a priority, intensity, frequency, or modality determined based on the fused information.

Context driven alerting

A pilot may be more stressed during take-off or landing, which is not abnormal. Physiological data of the pilot may be received. Placing the physiological data in context of the current situation may be advantageous in detecting anomalous behaviors of the pilot. A system and method are described. The system and method receive a stream of images from a camera and detect whether the pilot is exhibiting anomalous behavior. The anomalous behavior is further put into context based on the flight state and various avionics information.

Detecting anomalous behaviors within aircraft context

A system for evaluating communicative acts to determine a social network perception representation based on communicative acts between a plurality of communicative pairs where each communicative pair includes a first group and an individual or a second group participating in corresponding communicative acts. The system includes a patterns database, a scoring processor, and a communications network aggregation processor. The patterns database stores behavior recognition patterns defining particular behaviors and a score associated with each of the particular behaviors. The scoring processor is configured to determine a pairwise social perception score for each of the communicative pairs based on the particular behaviors identified in the communicative acts. The scoring processor is configured to determine a different pairwise social perception score for each of the communicative pairs. The communications network aggregation processor combines the pairwise social perception scores for each of the plurality of communicative pairs into a social perception network representation.

Systems and methods for determining social perception

A system for evaluating communicative acts, between a plurality of communicative pairs, to determine a social network perception representation. Each communicative pair includes two entities participating in a corresponding one or more of the communicative acts. The system includes a patterns database, a scoring processor, and a communications network aggregation processor. The patterns database stores behavior recognition patterns defining particular behaviors that may be used in the communicative acts and a social perception score associated with each of the particular behaviors. The communicative acts include non-verbal communicative acts. The scoring processor is configured to determine a pairwise social perception score for each communicative pair based on the particular behaviors identified in the communicative acts. The scoring processor is configured to determine a different pairwise social perception score for each of the communicative pairs. The communications network aggregation processor combines the pairwise social perception scores into a social perception network representation.

A system for evaluating communicative acts between communicative pairs to determine a social perception network representation. Each communicative pair includes two entities participating in a communicative act. The system includes a patterns database storing behavior recognition patterns defining particular behaviors that may be used in the communicative acts and a social perception score associated with each of the particular behaviors. The scoring processor determines a first pairwise social perception score for each communicative pair based on the particular behaviors in the communicative acts occurring during a first time frame. The scoring processor determines a different first pairwise social perception score for each of the communicative pairs during a first time frame and determines a different second social perception score for each of the communicative pairs during a second time frame. The communications network aggregation processor combines the pairwise social perception scores into a first social perception network representation.

Growing community of inventors

Ellicott City, MD, United States of America

Peggy Wu