The Boeing Company

Daniel Michael O'Shea

Shawn M Chamberlain

Deepak Khosla

Justin Cleve Hatcher

Jung Soon Jang

Leon Nguyen

Fan Hin Hung

Haden Harrison Smith

Joshua Neighbor

Disclosed herein is methods, systems, and aircraft for performing image analysis for aiding refueling operations. A tanker aircraft includes a single camera, a refueling boom, a camera configured to generate a 2D image of the refueling boom, a processor, and non-transitory computer readable storage media storing code. The code being executable by the processor to perform operations comprising receiving the 2D image from the single camera, determining 2D keypoints of the refueling boom located within the 2D image based on a predefined point model of the refueling boom, determining keypoints in 3D space based on the 2D keypoints to produce 3D keypoints, determining a 6DOF pose using the 2D keypoints and the 3D keypoints, and estimating a position of a tip of the refueling boom based on the 6DOF pose.

Systems and methods for calculating refueling tanker boom tip 3D position for aerial refueling

A trajectory planning system for a refueling boom includes a human-machine interface ('HMI') device and an electronic control unit (“ECU”). The HMI device outputs electronic control signals, in response to which the ECU performs a method. The ECU accesses a three-dimensional (“3D”) boundary model of the receiver and a 3D model of the boom. The ECU calculates a boom-to-receiver relative position using the models and sensor data, and a planned trajectory between a boom tip and a receptacle on the receiver. The trajectory is calculated using the boom-to-receiver relative position and predictive artificial potential fields. A point is found on a baseline trajectory farthest from a straight line between the receptacle and boom tip, which is recorded as a temporary goal. The planned trajectory avoids contact between the boom and receiver features. The ECU executes a control action using the planned trajectory.

Artificial potential field-based boom guidance for automated air-to-air refueling

An automated air-to-air refueling (A3R) system is usable with a tanker aircraft having a refueling boom. The system includes a camera connected to the tanker in proximity to the refueling boom which outputs a video stream of the boom and a fuel-receiving aircraft/receiver during an aerial refueling process. The system also includes a human-machine interface ('HMI') located aboard the tanker, and an electronic control unit (“ECU”) in communication with the camera and HMI. The ECU identifies keypoints on the receiver indicative of flight control surfaces thereof, tracks corresponding positions of the flight control surfaces in real-time, and predicts a change in position of the receiver in free space as a predicted 3D position using the corresponding positions. The HMI also outputs a directional indicator indicative of the predicted 3D position, e.g., as a graphical overlay to a display screen.

Automated air-to-air refueling using keypoint-based perception of aircraft flight control surfaces

Growing community of inventors

Seattle, WA, United States of America

Daniel Michael O'Shea