Amazon Technologies, Inc.

Microsoft Technology Licensing, LLC

Chengwu Luke Cui

Joshua John Watson

Barry James O'Brien

Scott Michael Wilcox

Sarah Graber

Scott Patrick Boyd

Benjamin Martin Schweitzer

Jean-Guillaume Dominique Durand

Gabor Szedo Becker

Yves Albers Schoenberg

Stephen Thomas Safarik

Anirudth Nambirajan

Systems and methods to detect objects and associated properties may be performed by an aerial vehicle having one or more propellers and one or more microphones. The aerial vehicle may emit propeller noise patterns via the propellers during operation, and the aerial vehicle may receive echoes of the propeller noise patterns via the microphones. Based on the emitted noise patterns and received echoes, the aerial vehicle may detect objects and associated properties within an environment of the aerial vehicle. In addition, the aerial vehicle may emit encoded propeller noise patterns via the propellers during operation to communicate with other aerial vehicles, and other aerial vehicles may receive the encoded propeller noise patterns via microphones. Using such encoded propeller noise patterns, a plurality of aerial vehicles may communicate and/or coordinate operations with each other.

Object detection using propeller noise

In some embodiments, a first image may be captured from a first field of view using a first exposure time. A second image may be captured from a second field of view using a second exposure time that is different than the first exposure time. An overlapping field of view may be defined by an overlapping portion of the first field of view and the second field of view. Histograms may be created for the first image and the second image, and possibly more images that include different exposure times and represent the overlapping field of view. The histograms may be analyzed to determine a presence or an absence of a light source in the overlapping field of view.

Light source detection in field of view

A system for detecting transparent elements in a vehicle environment are described. In some examples, this may include accessing an image of a scene captured by an image capture device attached to a vehicle. A reflected image present in the image may be detected. The reflected image may include a portion of the vehicle. It may be determined that the scene includes a transparent element based at least in part on detecting the reflected image present in the image.

Transparent element detection for vehicles

An aerial vehicle is outfitted with two or more digital cameras that may be continuously calibrated. The aerial vehicle may include one or more markings provided on portions of a frame, a control surface or another component that are within fields of view of the digital cameras. Based on an initial calibration of the digital cameras, intrinsic and extrinsic parameters of the digital cameras may be determined. Using the intrinsic and extrinsic parameters, locations of the markings may be determined. Subsequently, after the aerial vehicle has engaged in flight operations, images captured of the marking may be used to determine whether the digital cameras remain properly calibrated, and whether images captured using the imaging devices may be relied upon in determining ranges to objects or for any other purpose. A correction factor may be generated based on any differences between projections of the markings within the respective images.

Self-calibrating stereo camera pairs provided aboard aerial vehicles

Techniques for verifying a location and identification of a landing marker to aid an unmanned aerial vehicle (UAV) to deliver a payload to a location may be provided. For example, upon receiving an indication that a UAV has arrived to a delivery location, a server computer may process one or more images of an area that are provided by the UAV and/or a user interacting with a user device. A landing marker may be identified in the image and a representation of the landing marker along with instructions to guide the UAV to deliver the payload to the landing marker may be transmitted to the UAV and implemented by the UAV.

Drone marker and landing zone verification

Camera calibration may be performed in a mobile environment. One or more cameras can be mounted on a mobile vehicle, such as an unmanned aerial vehicle (UAV) or an automobile. Because of the mobility of the vehicle the one or more cameras may be subjected to inaccuracy in imagery caused by various factors, such as environmental factors (e.g., airflow, wind, etc.), impact by other objects (e.g., debris, vehicles, etc.), vehicle vibrations, and the like. To reduce the inaccuracy in imagery, the mobile vehicle can include a mobile camera calibration system configured to calibrate the one or more cameras while the mobile vehicle is traveling along a path. The mobile camera calibration system can cause the one or more cameras to capture an image of an imaging target while moving, and calibrate the one or more cameras based on a comparison between the image and imaging target data.

Camera calibration in a mobile environment

Systems and corresponding methods for monitoring of a state of charge or a state of health of a battery are described. For example, the battery monitoring system may include strain gauges to measure dimensions or changes in dimension of cells of the battery, and may determine states of charge of the cells based on dimensional changes over time, e.g., based on data from strain gauges. The dimensional changes may also be corrected for dimensional changes due to thermal strain. In addition, states of health of individual cells of the battery may be determined by comparing the dimensional changes of each of the cells over time. Further, the data from strain gauges may be received at various frequencies, e.g., based on an expected operational duration of the battery and/or based on a current state of charge or state of health.

Power supply monitoring system using strain gauges

Described is an imaging component that utilizes two rolling shutter sensors for motion detection of objects and for depth mapping of objects within an effective field of view of the imaging component. Unlike traditional stereo cameras that utilize global shutter sensors to avoid distortions, or attempting to remove distortions created by rolling shutter sensors, the disclosed implementations emphasize the distortions created by rolling shutters imaging moving objects and utilize that information to determine that the objects are moving and/or to determine a range or distance of the object from the imaging component. For example, a first rolling shutter sensor is oriented in a first orientation such that the scanlines generate the image from a top of the sensor to the bottom of the sensor, and a second rolling shutter sensor is oriented in a second orientation such that the scanlines generate the image from a bottom of the sensor to the top of the sensor. In other words, the two rolling shutter sensors have opposite orientations in the imaging component. As a result, the distortions of a moving object imaged by the two sensors are opposite and easily distinguished between the two images.

Rolling shutter motion and depth sensing

Systems and methods for detecting contamination on a sensor cover. In an example, an image sensor of a vehicle may be used to capture an image of an environment of the vehicle. A sensor cover may be positioned outwardly from the image sensor and may form a covered volume around at least a sensing surface of the image sensor. A light source may be positioned in the covered volume. A management module may be coupled with the image sensor. The management module may be configured to receive test image data of the environment generated by the image sensor while the light source is on, compare the test image data to baseline image data generated while the light source is off, and detect whether contamination exists on a surface of the sensor cover based on the comparison.

Vehicle camera contamination detection

Described herein are systems and methods of determining a distance between an object depicted in an image and an imaging device that captured that image. In particular, the disclosure discusses that an image may be separated into multiple derivative images, each of which is associated with a different wavelength of light. In some embodiments, an image may be separated into images associated with wavelengths of primary colors (e.g., red, green, and blue). Once separate images have been created, a sharpness value may be determined for each image. A distance between the object and the imaging device may then be calculated based on sharpness values associated with each of the separate images.

Distance detection based on chromatic aberration

This disclosure is directed to monitoring exposure levels for individual cameras of a stereo camera system to reduce unwanted lens flaring which may disrupt stereo vision capabilities of an unmanned aerial vehicle (UAV). The UAV may identify lens flaring by monitoring histogram data received from the cameras and then mitigate the lens flaring by performing aerial maneuvers with respect to the light source and/or moving camera componentry such as deploying a lens hood. The UAV may also identify lens flaring patterns that on a camera's sensor and determine a location of a peripheral light source based on these patterns. The UAV may then deploy a visor between the location of the peripheral light source and the cameras.

Flare detection and avoidance in stereo vision systems

Images captured by a camera system of an unmanned aerial vehicle (UAV) can be used to determine a weather condition in an environment of the UAV. The camera system of the UAV can capture one or more images, and a characteristic of at least one image of the one or more images can be determined from image data associated with the at least one image. A weather condition of the environment of the UAV can be determined based at least in part on the characteristic of the at least one image.

Image-based weather condition detection

A location marker that may be used to provide information to a vehicle, such as an unmanned aerial vehicle (UAV). The location marker may include a plurality of retroreflectors that may form a pattern readable by a vehicle or other device. The pattern may be read to extract an identifier of a location, such as an address or an identifier of a person. A global positioning system (GPS) device may transmit a general location of the location marker to the vehicle, while the location marker may provide a unique visual location identifier to a device within visual range of the location marker. In some embodiments, the location marker may also include lights that may be individually sequenced on and off at different times to create a time domain signal sequence that is readable by the vehicle.

Location marker with retroreflectors

Systems and corresponding methods for optical monitoring of a state of charge or a state of health of a battery are described. For example, the battery monitoring system may include an imaging device that captures images of cells of the battery and determines states of charge of the cells based on observed dimensional changes over time, e.g., relative to reference markers. The observed dimensional changes may also be corrected for dimensional changes due to thermal strain. In addition, states of health of individual cells of the battery may be determined by comparing the dimensional changes of each of the cells over time. Further, the images may be captured at various frequencies, e.g., based on an expected operational duration of the battery and/or based on a current state of charge or state of health.

Power supply monitoring system using optical estimation

Techniques for providing an object awareness guidance to clear a landing space may be provided. For example, during delivery an unmanned aerial vehicle (UAV) may capture an image of a potential landing zone and identify one or more objects in the image that may impede or obstruct delivery of the item in the potential landing zone. The UAV may be configured to generate and provide instructions to a user device to move or remove the identified one or more objects from the potential landing zone thereby creating a safe and unobstructed landing zone to deliver the item.

Growing community of inventors

Redmond, WA, United States of America

Chengwu Luke Cui