The Boeing Company

Hrl Laboratories, LLC

Leon Nguyen

Deepak Khosla

Fan Hin Hung

Haden Harrison Smith

Justin Cleve Hatcher

William K Leach

Yang Chen

Nick Shadbeh Evans

Jung Soon Jang

Paul S Idell

James L Clayton

Yifan Yang

Michelle D Warren

Shawn M Chamberlain

Daniel Michael O'Shea

Luis-Alberto J Santiago

Taraneh Sadjadpour

William Kentarou Kaneko Leach

Neale Ratzlaff

Tameez Latib

Methods, apparatuses and systems of training a neural network using in a vision-based tracking system are disclosed. A two-dimensional (2D) image of at least a portion of an object is received via a camera that is fixed, relative to the object. Subsequently, a keypoint detector predicts a set of keypoints on the object in the 2D image, generating predicted 2D keypoints. These predicted 2D keypoints are then projected into three-dimensional (3D) space, and keypoint depth information is added to generate predicted 3D keypoints. To enhance the training process, a 3D model of the object is utilized. Known rotational and translational information of the object in the 2D image is incorporated to known 3D model keypoints, resulting in transformed 3D model keypoints. Following this, a comparison between predicted 3D keypoints and transformed 3D model keypoints is made to calculate a loss value. The training process is further refined using an optimizer, minimizing the loss value during a training period.

Method of training a neural network for vision-based tracking and associated apparatus and system

Disclosed herein is methods, systems, and aircraft for performing image analysis for aiding refueling operations. A tanker aircraft includes a single camera, a refueling boom, a camera configured to generate a 2D image of the refueling boom, a processor, and non-transitory computer readable storage media storing code. The code being executable by the processor to perform operations comprising receiving the 2D image from the single camera, determining 2D keypoints of the refueling boom located within the 2D image based on a predefined point model of the refueling boom, determining keypoints in 3D space based on the 2D keypoints to produce 3D keypoints, determining a 6DOF pose using the 2D keypoints and the 3D keypoints, and estimating a position of a tip of the refueling boom based on the 6DOF pose.

Systems and methods for calculating refueling tanker boom tip 3D position for aerial refueling

A system for refining a six degrees of freedom pose estimate of a target object based on a one-dimensional measurement includes a camera and a range-sensing device. The range-sensing device is configured to determine an actual distance measured between the range-sensing device and an actual point of intersection. The range-sensing device projects a line-of-sight that intersects with the target object at the actual point of intersection. The system also includes one or more processors in electronic communication with the camera and the range-sensing device and a memory coupled to the processors. The memory stores data into one or more databases and program code that, when executed by the processors, causes the system to predict the six degrees of freedom pose estimate of the target object. The system also determines a revised six degrees of freedom pose estimate of the target object based on at least an absolute error.

System for refining a six degrees of freedom pose estimate of a target object

Aspects of the disclosure provide fuel receptacle position estimation for aerial refueling (derived from aircraft position estimation). A video stream comprising a plurality of video frames each showing an aircraft to be refueled, is received from a single camera. An initial position estimate is determined for the aircraft for the plurality of video frames, generating an estimated flight history for the aircraft. The estimated flight history for the aircraft is used to determine a temporally consistent refined position estimate, based on known aircraft flight path trajectories in an aerial refueling setting. The position of a fuel receptacle on the aircraft is determined, based on the refined position estimate for the aircraft, and an aerial refueling boom may be controlled to engage the fuel receptacle. Examples may use a deep learning neural network (NN) or optimization (e.g., bundle adjustment) to determine the refined position estimate from the estimated flight history.

Temporally consistent position estimation refinement for aerial refueling

Aspects of the disclosure provide fuel receptacle and boom tip position and pose estimation for aerial refueling. A video frame is received and within the video frame, aircraft keypoints for an aircraft to be refueled are determined. Based on at least the aircraft keypoints, a position and pose of a fuel receptacle on the aircraft is determined. Within the video frame, a boom tip keypoint for a boom tip of an aerial refueling boom is also determined. Based on at least the boom tip keypoint, a position and pose of the boom tip is determined. Based on at least the position and pose of the fuel receptacle and the position and pose of the boom tip, the aerial refueling boom is controlled to engage the fuel receptacle. Some examples overlay projections of an aircraft model on displayed video for a human observer.

Fuel receptacle and boom tip position and pose estimation for aerial refueling

Aspects of the disclosure provide fuel receptacle position/pose estimation for aerial refueling (derived from aircraft position and pose estimation). A video frame, showing an aircraft to be refueled, is received from a single camera. An initial position/pose estimate is determined for the aircraft, which is used to generating an initial rendering of an aircraft model. The video frame and the initial rendering are used to determining refinement parameters (e.g., a translation refinement and a rotational refinement) for the initial position/pose estimate, providing a refined position/pose estimate for the aircraft. The position/pose of a fuel receptacle on the aircraft is determined, based on the refined position/pose estimate for the aircraft, and an aerial refueling boom may be controlled to engage the fuel receptacle. Examples extract features from the aircraft in the video frame and the aircraft model rendering, and use a deep learning neural network (NN) to determine the refinement parameters.

Pose estimation refinement for aerial refueling

A method, apparatus and computer program product are provided to generate a model of one or more objects relative to a vehicle. In the context of a method, radar information is received in the form of in-phase quadrature (IQ) data and the IQ data is converted to one or more first range-doppler maps. The method further includes evaluating the one or more first range-doppler maps with a machine learning model to generate the model that captures the detection of the one or more objects relative to the vehicle. A corresponding apparatus and computer program product are also provided.

Radar-based method and apparatus for generating a model of an object relative to a vehicle

Described is a system for onboard, real-time, activity detection and classification. During operation, the system detects one or more objects in a scene using a mobile platform and tracks each of the objects as the objects move in the scene to generate tracks for each object. The tracks are transformed using a fuzzy-logic based mapping to semantics that define group activities of the one or more objects in the scene. Finally, a state machine is used to determine whether the defined group activities are normal or abnormal phases of a predetermined group operation.

Growing community of inventors

Tarzana, CA, United States of America

Leon Nguyen