Hrl Laboratories, LLC

The Boeing Company

Raytheon Company

Xlabs Holdings, LLC

Hrl Labortories, LLC

Theboeing Company

Deepak Khosla

Yang Chen

David J Huber

Kyungnam Kim

Lei Zhang

Ryan M Uhlenbrock

Shinko Yuanhsien Cheng

Changsoo S Jeong

Alexander L Honda

Kevin R Martin

Sean Soleyman

Fan Hin Hung

Fredy Monterroza

Leon Nguyen

Jiejun Xu

Rajan Bhattacharyya

Yongqiang Cao

Haden Harrison Smith

Paul Alex Dow

Yuri Owechko

Narayan Srinivasa

Swarup S Medasani

Anthony Wayne Baker

Brandon M Courter

Joshua G Fadaie

Darrel J VanBuer

James Guillochon

Michael J Daily

Suhas E Chelian

Shane Edward Arthur

Justin Cleve Hatcher

Matthew S Keegan

Zhichao Chen

David L Ii

William K Leach

Charles Richard Tullock

Tsai-Ching Lu

David L Allen

Qin Jiang

Nick Shadbeh Evans

Aleksey Nogin

Nigel D Stepp

Terrell Nathan Mundhenk

Jung Soon Jang

Ronald Tadao Azuma

Luke Charles Ingram

Eric Huang

Samuel D Johnson

Michael A Warren

Howard C Choe

Paul S Idell

Navid Naderializadeh

Yifan Yang

James L Clayton

Michelle D Warren

Shawn M Chamberlain

Thomas S Nichols

Daniel Michael O'Shea

Ignacio M Soriano

Luis-Alberto J Santiago

Jeremy Daniel Howe

David A Fuciarelli

Scott Rad

Richard Hanes

Chris A Cantrell

Taraneh Sadjadpour

Aaron Feldman

Jacob Moorman

William Kentarou Kaneko Leach

Benjamin Nuernberger

Anders Eirik Heussy

Fan H Hung

Shawn Moffit

Calvin Chung

Penn Tasinga

Christopher Kanan

Alex Dow

Huapeng Su

Chun Kit Chung

Kevin Martin

Disclosed herein is methods, systems, and aircraft for performing image analysis for aiding refueling operations. A tanker aircraft includes a single camera, a refueling boom, a camera configured to generate a 2D image of the refueling boom, a processor, and non-transitory computer readable storage media storing code. The code being executable by the processor to perform operations comprising receiving the 2D image from the single camera, determining 2D keypoints of the refueling boom located within the 2D image based on a predefined point model of the refueling boom, determining keypoints in 3D space based on the 2D keypoints to produce 3D keypoints, determining a 6DOF pose using the 2D keypoints and the 3D keypoints, and estimating a position of a tip of the refueling boom based on the 6DOF pose.

Systems and methods for calculating refueling tanker boom tip 3D position for aerial refueling

A method of monitoring an assembly process is disclosed herein. The method includes obtaining an event model for each of a plurality of objects in the assembly process with the event model for each of the plurality of objects including a predetermined time frame for a change in presence to occur. The method includes collecting an image sequence of the assembly process for monitoring and identifying if a change in presence for each of the plurality of objects occurred with a detector model. The method further includes reviewing the event model for each of the plurality of objects to determine if the predetermined time frame lapsed without the change in presence of a corresponding one of the plurality of objects being identified and issuing an alert if the predetermined time frame lapsed without the presence of a corresponding one of the plurality of objects being identified.

Method and system for process monitoring

A vehicle management system comprising a computer system and an agent. The agent comprises a machine learning model and a rule system. The machine learning model system is trained to receive observations for the vehicle system and select a behavior for the vehicle system in response to receiving the observations. The rule system is configured to select a set of actions to execute the behavior for the vehicle system in response to a selection of the behavior by the machine learning model system.

Behavior management of multiple vehicles

Training adversarial aircraft controllers is provided. The method comprises inputting current observed states of a number of aircraft into a world model encoder, wherein each current state represents a state of a different aircraft, and wherein each current state comprises a missing parameter value. A number of adversarial control actions for the aircraft are input into the world model encoder concurrently with the current observed state, wherein the adversarial control actions are generated by competing neural network controllers. The world model encoder generates a learned observation from the current observed states and adversarial control actions, wherein the learned observation represents the missing parameter value from the current observed states. The learned observation and current observed states are input into the competing neural network controllers, wherein each current observed state is fed into a respective controller. The competing neural network controllers then generate next adversarial control actions.

Predictive modeling of aircraft dynamics

Described is a learning system for multi-agent applications. In operation, the system initializes a plurality of learning agents. The learning agents include both tactical agents and strategic agents. The strategic agents take an observation from an environment and select one or more of the tactical agents to produce an action that is used to control a platform's actuators or simulated movements in the environment to complete a task. Alternatively, the tactical agents produce the action corresponding to a learned low-level behavior to control the platform's actuators or simulated movements in the environment to complete the task.

Robust, scalable and generalizable machine learning paradigm for multi-agent applications

A system for refining a six degrees of freedom pose estimate of a target object based on a one-dimensional measurement includes a camera and a range-sensing device. The range-sensing device is configured to determine an actual distance measured between the range-sensing device and an actual point of intersection. The range-sensing device projects a line-of-sight that intersects with the target object at the actual point of intersection. The system also includes one or more processors in electronic communication with the camera and the range-sensing device and a memory coupled to the processors. The memory stores data into one or more databases and program code that, when executed by the processors, causes the system to predict the six degrees of freedom pose estimate of the target object. The system also determines a revised six degrees of freedom pose estimate of the target object based on at least an absolute error.

System for refining a six degrees of freedom pose estimate of a target object

Described is a system for performing a simulated vehicle control task based on adversarial decision analysis. Empirical game theoretic analyses are performed between an evolving and an adversary population of neural network strategies. Each empirical game theoretic analysis includes using a neuroevolution procedure to perform a fitness-based selection of a strategy in the evolving population that out-performs the adversary population. Using an empirical game theory procedure, the neuroevolution procedure is iteratively run and the selected strategy is added to the adversary population with each iteration, resulting in monotonic strategy improvement with each iteration. Following the empirical game theoretic analyses, a final strategy is selected for the evolving population and the adversary population using a tournament selection procedure. The final strategy is used to train a neural network which is used to perform a simulated vehicle control task.

Empirical game theoretic system and method for adversarial decision analysis

Aspects of the disclosure provide fuel receptacle position estimation for aerial refueling (derived from aircraft position estimation). A video stream comprising a plurality of video frames each showing an aircraft to be refueled, is received from a single camera. An initial position estimate is determined for the aircraft for the plurality of video frames, generating an estimated flight history for the aircraft. The estimated flight history for the aircraft is used to determine a temporally consistent refined position estimate, based on known aircraft flight path trajectories in an aerial refueling setting. The position of a fuel receptacle on the aircraft is determined, based on the refined position estimate for the aircraft, and an aerial refueling boom may be controlled to engage the fuel receptacle. Examples may use a deep learning neural network (NN) or optimization (e.g., bundle adjustment) to determine the refined position estimate from the estimated flight history.

Temporally consistent position estimation refinement for aerial refueling

Aspects of the disclosure provide fuel receptacle and boom tip position and pose estimation for aerial refueling. A video frame is received and within the video frame, aircraft keypoints for an aircraft to be refueled are determined. Based on at least the aircraft keypoints, a position and pose of a fuel receptacle on the aircraft is determined. Within the video frame, a boom tip keypoint for a boom tip of an aerial refueling boom is also determined. Based on at least the boom tip keypoint, a position and pose of the boom tip is determined. Based on at least the position and pose of the fuel receptacle and the position and pose of the boom tip, the aerial refueling boom is controlled to engage the fuel receptacle. Some examples overlay projections of an aircraft model on displayed video for a human observer.

Fuel receptacle and boom tip position and pose estimation for aerial refueling

Apparatuses, systems, and methods dynamically model intrinsic parameters of a camera. Methods include: collecting, using a camera having a focus motor, calibration data at a series of discrete focus motor positions; generating, from the calibration data, a set of constant point intrinsic parameters; determining, from the set of constant point intrinsic parameters, a subset of intrinsic parameters to model dynamically; performing, for each intrinsic parameter of the subset of intrinsic parameters, a fit of the point intrinsic parameter values against focus motor positions; generating a model of the intrinsic parameters for the camera based, at least in part, on the fit of the point intrinsic parameter values against the focus motor positions; and determining a position of a fiducial marker within a field of view of the camera based, at least in part, on the model of the intrinsic parameters for the camera.

Method and apparatus for modeling dynamic intrinsic parameters of a camera

Aspects of the disclosure provide fuel receptacle position/pose estimation for aerial refueling (derived from aircraft position and pose estimation). A video frame, showing an aircraft to be refueled, is received from a single camera. An initial position/pose estimate is determined for the aircraft, which is used to generating an initial rendering of an aircraft model. The video frame and the initial rendering are used to determining refinement parameters (e.g., a translation refinement and a rotational refinement) for the initial position/pose estimate, providing a refined position/pose estimate for the aircraft. The position/pose of a fuel receptacle on the aircraft is determined, based on the refined position/pose estimate for the aircraft, and an aerial refueling boom may be controlled to engage the fuel receptacle. Examples extract features from the aircraft in the video frame and the aircraft model rendering, and use a deep learning neural network (NN) to determine the refinement parameters.

Pose estimation refinement for aerial refueling

Apparatuses and methods train a model and then use the trained model to determine a global three dimensional (3D) position and orientation of a fiduciary marker. In the context of an apparatus for training a model, a wider field-of-view sensor is configured to acquire a static image of a space in which the fiducial marker is disposed and a narrower field-of-view sensor is configured to acquire a plurality of images of at least a portion of the fiducial marker. The apparatus also includes a pan-tilt unit configured to controllably alter pan and tilt angles of the narrower field-of-view sensor during image acquisition. The apparatus further includes a control system configured to determine a transformation of position and orientation information determined from the images acquired by the narrower field-of-view sensor to a coordinate system for the space for which the static image is acquired by the wider field-of-view sensor.

Method and apparatus for determining a three-dimensional position and pose of a fiducial marker

A method, apparatus and computer program product are provided to generate a model of one or more objects relative to a vehicle. In the context of a method, radar information is received in the form of in-phase quadrature (IQ) data and the IQ data is converted to one or more first range-doppler maps. The method further includes evaluating the one or more first range-doppler maps with a machine learning model to generate the model that captures the detection of the one or more objects relative to the vehicle. A corresponding apparatus and computer program product are also provided.

Radar-based method and apparatus for generating a model of an object relative to a vehicle

Described is a system for controlling multiple autonomous platforms. A training process is performed to produce a trained learning agent in a simulation environment. In each episode, each controlled platform is assigned to one target platform that produces an observation. A learning agent processes the observation using a deep learning network and produces an action corresponding to each controlled platform until an action has been produced for each controlled platform. A reward value is obtained corresponding to the episode. The trained learning agent is executed to control each autonomous platform, where the trained agent receives one or more observations from one or more platform sensors and produces an action based on the one or more observations. The action is then used to control one or more platform actuators.

Multi-agent planning and autonomy

Apparatuses and methods determine the three-dimensional position and orientation of a fiducial marker and tracking the three-dimensional position and orientation across different fields-of-view. Methods include: capturing an image of a first space in which the fiducial marker is disposed with a first sensor having a first field-of-view; determining the three-dimensional location and orientation of the fiducial marker within the first space based on the image of the first space in which the fiducial marker is disposed; capturing an image of a second space in which the fiducial marker is disposed with a second sensor having a second field-of-view; calculating pan and tilt information for the second sensor to move the second field-of-view of the second sensor to acquire an image of the fiducial marker; and determining the three-dimensional location and orientation of the fiducial marker within the second space based on the image of the second space.

Method and apparatus for hand-off and tracking for pose estimation of a fiducial marker

Growing community of inventors

Camarillo, CA, United States of America

Deepak Khosla