The Boeing Company

Fan Hin Hung

Deepak Khosla

Leon Nguyen

Haden Harrison Smith

Yang Chen

Justin Cleve Hatcher

Sean Soleyman

Jung Soon Jang

Navid Naderializadeh

Paul S Idell

Yifan Yang

James L Clayton

Joshua G Fadaie

Shawn M Chamberlain

Daniel Michael O'Shea

William K Leach

Luis-Alberto J Santiago

Charles Richard Tullock

Taraneh Sadjadpour

William Kentarou Kaneko Leach

Neale Ratzlaff

Disclosed herein is methods, systems, and aircraft for performing image analysis for aiding refueling operations. A tanker aircraft includes a single camera, a refueling boom, a camera configured to generate a 2D image of the refueling boom, a processor, and non-transitory computer readable storage media storing code. The code being executable by the processor to perform operations comprising receiving the 2D image from the single camera, determining 2D keypoints of the refueling boom located within the 2D image based on a predefined point model of the refueling boom, determining keypoints in 3D space based on the 2D keypoints to produce 3D keypoints, determining a 6DOF pose using the 2D keypoints and the 3D keypoints, and estimating a position of a tip of the refueling boom based on the 6DOF pose.

Systems and methods for calculating refueling tanker boom tip 3D position for aerial refueling

A vehicle management system comprising a computer system and an agent. The agent comprises a machine learning model and a rule system. The machine learning model system is trained to receive observations for the vehicle system and select a behavior for the vehicle system in response to receiving the observations. The rule system is configured to select a set of actions to execute the behavior for the vehicle system in response to a selection of the behavior by the machine learning model system.

Behavior management of multiple vehicles

Training adversarial aircraft controllers is provided. The method comprises inputting current observed states of a number of aircraft into a world model encoder, wherein each current state represents a state of a different aircraft, and wherein each current state comprises a missing parameter value. A number of adversarial control actions for the aircraft are input into the world model encoder concurrently with the current observed state, wherein the adversarial control actions are generated by competing neural network controllers. The world model encoder generates a learned observation from the current observed states and adversarial control actions, wherein the learned observation represents the missing parameter value from the current observed states. The learned observation and current observed states are input into the competing neural network controllers, wherein each current observed state is fed into a respective controller. The competing neural network controllers then generate next adversarial control actions.

Predictive modeling of aircraft dynamics

A system for refining a six degrees of freedom pose estimate of a target object based on a one-dimensional measurement includes a camera and a range-sensing device. The range-sensing device is configured to determine an actual distance measured between the range-sensing device and an actual point of intersection. The range-sensing device projects a line-of-sight that intersects with the target object at the actual point of intersection. The system also includes one or more processors in electronic communication with the camera and the range-sensing device and a memory coupled to the processors. The memory stores data into one or more databases and program code that, when executed by the processors, causes the system to predict the six degrees of freedom pose estimate of the target object. The system also determines a revised six degrees of freedom pose estimate of the target object based on at least an absolute error.

System for refining a six degrees of freedom pose estimate of a target object

Aspects of the disclosure provide fuel receptacle position estimation for aerial refueling (derived from aircraft position estimation). A video stream comprising a plurality of video frames each showing an aircraft to be refueled, is received from a single camera. An initial position estimate is determined for the aircraft for the plurality of video frames, generating an estimated flight history for the aircraft. The estimated flight history for the aircraft is used to determine a temporally consistent refined position estimate, based on known aircraft flight path trajectories in an aerial refueling setting. The position of a fuel receptacle on the aircraft is determined, based on the refined position estimate for the aircraft, and an aerial refueling boom may be controlled to engage the fuel receptacle. Examples may use a deep learning neural network (NN) or optimization (e.g., bundle adjustment) to determine the refined position estimate from the estimated flight history.

Temporally consistent position estimation refinement for aerial refueling

Aspects of the disclosure provide fuel receptacle and boom tip position and pose estimation for aerial refueling. A video frame is received and within the video frame, aircraft keypoints for an aircraft to be refueled are determined. Based on at least the aircraft keypoints, a position and pose of a fuel receptacle on the aircraft is determined. Within the video frame, a boom tip keypoint for a boom tip of an aerial refueling boom is also determined. Based on at least the boom tip keypoint, a position and pose of the boom tip is determined. Based on at least the position and pose of the fuel receptacle and the position and pose of the boom tip, the aerial refueling boom is controlled to engage the fuel receptacle. Some examples overlay projections of an aircraft model on displayed video for a human observer.

Fuel receptacle and boom tip position and pose estimation for aerial refueling

Aspects of the disclosure provide fuel receptacle position/pose estimation for aerial refueling (derived from aircraft position and pose estimation). A video frame, showing an aircraft to be refueled, is received from a single camera. An initial position/pose estimate is determined for the aircraft, which is used to generating an initial rendering of an aircraft model. The video frame and the initial rendering are used to determining refinement parameters (e.g., a translation refinement and a rotational refinement) for the initial position/pose estimate, providing a refined position/pose estimate for the aircraft. The position/pose of a fuel receptacle on the aircraft is determined, based on the refined position/pose estimate for the aircraft, and an aerial refueling boom may be controlled to engage the fuel receptacle. Examples extract features from the aircraft in the video frame and the aircraft model rendering, and use a deep learning neural network (NN) to determine the refinement parameters.

Pose estimation refinement for aerial refueling

Growing community of inventors

Los Angeles, CA, United States of America

Fan Hin Hung