The Boeing Company

Eric Raymond Muir

Nick Shadbeh Evans

Nick Shabdeh Evans

Tyler Charles Staudinger

Kevin Scott Callahan

Joshua David Kalin

Cullen W Billhartz

Stephen G Dame

Zachary D Jorgensen

Isaac Chang

Jasper P Corleis

Michelle D Warren

Vandan Patel

Mary Karroqe

A method is provided for supporting an aircraft approaching a runway on an airfield. The method includes receiving a sequence of images of the airfield, captured by a camera onboard the aircraft approaching the runway. For at least one image of the sequence of images, the method includes applying the image(s) to a machine learning model trained to predict a pose of the aircraft relative to the runway. The machine learning model is configured to map the image(s) to the pose based on a training set of labeled images with respective ground truth poses of the aircraft relative to the runway. The pose is output as a current pose estimate of the aircraft relative to the runway for use in at least one of monitoring the current pose estimate, generating an alert based on the current pose estimate, or guidance or control of the aircraft on a final approach.

Determining a current pose estimate of an aircraft relative to a runway to support the aircraft on approach

Systems and methods of aligning removable sensors mounted on a vehicle based upon sensor output of such sensors are described. Sensor output is collected from a removable sensor (e.g., a digital camera), and a representation of such sensor output is generated (e.g., a digital image). The representation of the sensor output is compared against a spatial template (e.g., a digital mask overlaid on the representation) to determine whether external references in the representation align with corresponding reference indicators in the spatial template. When alignment is required, the removable sensor is aligned by one or both of the following until the external references in a representation of sensor data at an updated current location align with the corresponding reference indicators in the spatial template: (i) adjusting the position of the removable sensor on the vehicle or (ii) adjusting the representation of the sensor output to simulate such repositioning.

Aligning sensors on vehicles using sensor output

A computer-implemented method is provided for detecting a dynamic object in a visual line of sight of an aircraft. The method includes implementing, by a computer system, the following operations: receiving at least two image frames; detecting a horizon within at least one of the at least two image frames; segmenting each of the at least two image frames into at least first and second search regions using the detected horizon, wherein the first search region is defined above the detected horizon in the at least two image frames and the second search region is defined below the horizon in the at least two image frames; determining, across the at least two image frames, a motion vector field within at least one of the first search region or the second search region; and identifying a detection proposal using the determined motion vector field.

Method and system for aerial object detection

Solutions are provided for auto-labeling sensor data for machine learning (ML). An example includes: determining a platform's own position; recording, from a sensor aboard the platform, sensor data comprising a sensor image; receiving position data for at least one intruder object (e.g., a nearby airborne object); based at least on the position data for the intruder object and the platform's position, determining a relative position and a relative velocity of the intruder object; based at least on the relative position and a relative velocity of the intruder object and a field of view of the sensor, determining an expected position of the intruder object in the sensor image; labeling the sensor image, wherein the labeling comprises annotating the sensor image with a region of interest and an object identification; and training an artificial intelligence (AI) model using the labeled sensor image.

Auto-labeling sensor data for machine learning

A method of supporting an aircraft approaching a runway is provided. Example implementations involve receiving a sequence of images, captured by a camera onboard the aircraft. Example implementations involve applying a received image to a machine learning model trained to detect the runway or a runway marking in the image, and to produce a mask that includes a segment of pixels of the image assigned to an object class for the runway or marking. Example implementations may also involve applying the mask to a corner detector to detect interest points on the mask and match the interest points to corresponding points on the runway or the marking that have known runway-framed local coordinates. Example implementations may also involve performing a perspective-n-point estimation to determine a current pose estimate of the aircraft relative to the runway or the marking and outputting the current pose estimate for use during final approach.

Perception-based autonomous landing for aircraft

A method for aligning a removable sensor on a vehicle includes connecting the removable sensor to a sensor mounting device. The method further includes connecting a connector of an alignment apparatus to either (i) the removable sensor such that a spatial reference component of the alignment apparatus has a known position and orientation relative to a current position and orientation of the removable sensor or (ii) a fixed connection location on the vehicle such that the spatial reference component indicates a desired position and orientation of the removable sensor. In addition the method includes adjusting the current position and orientation of the removable sensor by reference to the alignment apparatus to cause the current position and orientation of the removable sensor to match the desired position and orientation of the removable sensor.

Method, system and apparatus for aligning a removable sensor on a vehicle

A computer-implemented method for generating a training set of images and labels for a native environment includes receiving physical coordinate sets, retrieving environmental model data corresponding to a georeferenced model of the environment, and creating a plurality of two-dimensional (2-D) rendered images each corresponding to a view from one of the physical coordinate sets. The 2-D rendered images include one or more of the environmental features. The method also includes generating linking data associating each of the 2-D rendered images with (i) labels for the one or more included environmental features and (ii) a corresponding native image. Additionally, the method includes storing the training set including the 2-D rendered images, labels, corresponding native images, and linking data.

Systems and methods for automatically generating training image sets for an environment

Systems, methods, and computer-readable media storing instructions for determining cross-track error of an aircraft on a taxiway are disclosed herein. The disclosed techniques capture electronic images of a portion of the taxiway using cameras or other electronic imaging devices mounted on the aircraft, pre-process the electronic images to generate regularized image data, apply a trained multichannel neural network model to the regularized image data to generate a preliminary estimate of cross-track error relative to the centerline of the taxiway, and post-process the preliminary estimate to generate an estimate of cross-track error of the aircraft. Further embodiments adjust a GPS-based location estimate of the aircraft using the estimate of cross-track error or adjust the heading of the aircraft based upon the estimate of cross-track error.

Aircraft positioning on a taxiway

A computer-implemented method for generating a training image set includes retrieving, from at least one memory device, model data corresponding to a three-dimensional (3-D) model of a target object, and creating a plurality of two-dimensional (2-D) synthetic images from the model data. The 2-D synthetic images include a plurality of views of the 3-D model. The method also includes creating a plurality of semantic segmentation images by identifying a plurality of pixels that define the target object in the 2-D synthetic image, and assigning a semantic segmentation label to the identified pixels of the target object. The method further includes generating linking data associating each of the semantic segmentation labels with a corresponding one of the 2-D synthetic images, and storing the training image set including the 2-D synthetic images, the semantic segmentation labels, and the linking data.

Growing community of inventors

Bothell, WA, United States of America

Eric Raymond Muir