Verily Life Sciences LLC

Wing Aviation LLC

Ali Shoeb

Anupam J Pathak

Marcus Hammond

Raphael Cendrillon

Christopher Cobar

Mark Murphy

Kevin Jenkins

Kyle Krafka

Kyle Julian

Fuad Al-Amin

Dinuka Abeywardena

Christopher Towles Lengerich

Domitille Commun

David Ben Shimol

Russell N Mirov

Emre Demiralp

Tushar Anil Parlikar

Ivan Qiu

Scott Brian Kennedy

Joshua Burkart

William Jacob Benhardt Biesinger

Nikhil Bikhchandani

John Mooney

Jeremie Gabor

Lance Jonathan Myers

Hongsheng Wang

David Stonestrom

A technique performed by UAV delivery system includes: arriving by a UAV of over a destination area; capturing a plurality of aerial images of a scene at the destination area with an onboard camera system of the UAV while flying above the destination area, wherein the aerial images capture the scene from a plurality of UAV vantage points offset from each other; optimizing weights of a generative neural network (GNN) using at least some of the aerial images as a training dataset to encode a volumetric representation of the scene into the GNN, wherein the weights are optimized by an onboard processing system of the UAV; and communicating the GNN with the weights optimized to a backend datacenter in communication with the UAV to transmit the volumetric representation of the scene over which the UAV flew without transmitting the aerial images themselves to the backend datacenter.

Compressing a scene into a generative neural network for UAV scene analysis applications

A technique for detection of an obstacle by a UAV includes arriving above a location at a first altitude by the UAV; navigating a descent flight pattern from the first altitude towards the location; acquiring aerial images of the location below the UAV with a camera system disposed onboard the UAV; and analyzing the aerial images with a machine vision system disposed onboard the UAV that is adapted to detect a presence of the obstacle in the aerial images. The descent flight pattern is selected to increase perception by the machine vision system of the obstacle.

Using UAV flight patterns to enhance machine vision detection of obstacles

A technique for detecting and avoiding obstacles by an unmanned aerial vehicle (UAV) includes: querying a knowledge graph having information related to a dynamic obstacle that may be in proximity to the UAV when traveling along a planned route; comparing the location of the dynamic obstacle to the UAV to detect conflicts; and in response to detecting a conflict, performing an action to avoid conflict with the dynamic obstacle. The knowledge graph can be updated by receiving a VHF radio signal containing the information related to the dynamic obstacle in the audible speech format; translating the audible speech format to a text format using speech recognition; analyzing the text format for relevant information related to the dynamic obstacle; comparing the relevant information related to the dynamic obstacle of the text format to the knowledge graph to detect changes; and updating the knowledge graph.

Autonomous detect and avoid from speech recognition and analysis

A technique for a UAV includes: acquiring an aerial image of an area below a UAV that includes one or more instances of an object; analyzing the aerial image with an image classifier to classify select pixels of the aerial image as being keypoint pixels associated with keypoints of the object; grouping the keypoint pixels into one or more groups each associated with one of the instances of the object, wherein first keypoint pixels of the keypoint pixels are grouped into a first group of the one or more groups associated with a first instance of the one or more instances of the object; generating an estimate of a relative position of the UAV to the first instance of the object based at least upon a machine vision analysis of the first keypoint pixels; and navigating the UAV into alignment with the first instance based upon the estimate.

Visual detection and localization of package autoloaders by UAV

A technique for validating a presence of a package carried by an unmanned aerial vehicle (UAV) includes: capturing an image of a scene below the UAV with a camera mounted to the UAV and oriented to face down from the UAV; analyzing the image to identify whether the package is present in the image; and determining whether the package is attached to the UAV, via a tether extending from an underside of the UAV, based at least on the analyzing of the image.

Visual and tactile confirmation of package presence for UAV aerial deliveries

A method includes navigating, by an uncrewed aerial vehicle (UAV), to a delivery location in an environment. The method also includes capturing, by at least one sensor on the UAV, sensor data representative of the delivery location. The method further includes determining, based on the sensor data representative of the delivery location, a segmented point cloud. The segmented point cloud defines a point cloud of the delivery location segmented into a plurality of point cloud areas with corresponding semantic classifications. The method additionally includes determining, based on the segmented point cloud, at least one delivery point in the delivery location. The at least one delivery point in the delivery location satisfies at least one condition indicating that a descent path above the at least one delivery point represented in the point cloud is at least a particular lateral distance away from point cloud areas with corresponding semantic classifications indicative of an obstacle at the delivery location. The method also includes transmitting, by the UAV, the at least one delivery point to a server device.

Automatic selection of delivery zones using survey flight 3D scene reconstructions

Techniques for identifying a close encounter between an aircraft and an obstacle are disclosed. The technique includes acquiring a video stream of a ground area below the aircraft with a stereovision camera system disposed onboard the aircraft. A depth perception map is generated with a stereovision processing pipeline indicating stereovision depth estimates of first image pixels from the video stream. An optical flow map is generated with an optical flow processing pipeline indicating optical flow depth estimates of second image pixels from the video stream. The depth perception and optical flow maps are compared. An encounter flag indicating that the close encounter between the aircraft and the obstacle occurred is asserted based at least on the comparing.

Detection of close encounters with obstacles by aerial vehicles

A technique for avoiding obstacles by an unmanned aerial vehicle (UAV) includes: acquiring an aerial image of a ground area below the UAV; analyzing the aerial image to identify a shadow in the aerial image cast by an object rising from the ground area; determining a pixel length of the shadow in the aerial image; calculating an estimated height of the object based at least on the pixel length of the shadow and an angle of the sun when the aerial image is acquired; and generating a clearance zone around the object having at least one dimension determined based on the estimated height, wherein the clearance zone represents a region in space to avoid when navigating the UAV.

Obstacle avoidance for aircraft from shadow analysis

A method includes receiving a two-dimensional (2D) image captured by a camera on a unmanned aerial vehicle (UAV) and representative of an environment of the UAV. The method further includes applying a trained machine learning model to the 2D image to produce a semantic image of the environment and a depth image of the environment, where the semantic image comprises one or more semantic labels. The method additionally includes retrieving reference depth data representative of the environment, wherein the reference depth data includes reference semantic labels. The method also includes aligning the depth image of the environment with the reference depth data representative of the environment to determine a location of the UAV in the environment, where the aligning associates the one or more semantic labels from the semantic image with the reference semantic labels from the reference depth data.

Machine-learned monocular depth estimation and semantic segmentation for 6-DOF absolute localization of a delivery drone

A method includes capturing, by a sensor on an unmanned aerial vehicle (UAV), an image of a delivery location. The method also includes determining, based on the image of the delivery location, a segmentation image. The segmentation image segments the delivery location into a plurality of pixel areas with corresponding semantic classifications. The method additionally includes determining, based on the segmentation image, a percentage of obstacle pixels within a surrounding area of a delivery point at the delivery location, wherein each obstacle pixel has a semantic classification indicative of an obstacle in the delivery location. The method further includes based on the percentage of obstacle pixels being above a threshold percentage, aborting a delivery process of the UAV.

Semantic abort of unmanned aerial vehicle deliveries

In some embodiments, an unmanned aerial vehicle (UAV) is provided. The UAV comprises one or more processors; a camera; one or more propulsion devices; and a computer-readable medium having instructions stored thereon that, in response to execution by the one or more processors, cause the UAV to perform actions comprising: receiving at least one image captured by the camera; generating labels for pixels of the at least one image by providing the at least one image as input to a machine learning model; identifying one or more landing spaces in the at least one image based on the labels; determining a relative position of the UAV with respect to the one or more landing spaces; and transmitting signals to the one or more propulsion devices based on the relative position of the UAV with respect to the one or more landing spaces.

Pixel-by-pixel segmentation of aerial imagery for autonomous vehicle control

A method includes capturing, by a sensor on an unmanned aerial vehicle (UAV), an image of a delivery location. The method further includes determining, based on the image of the delivery location, a segmentation image. The segmentation image segments the delivery location into a plurality of pixel areas with corresponding semantic classifications. The method also includes determining, based on the segmentation image, a distance-to-obstacle image of a delivery zone at the delivery location. The distance-to-obstacle image comprises a plurality of pixels, each pixel representing a distance in the segmentation image from a nearest pixel area with a semantic classification indicative of an obstacle in the delivery location. Additionally, the method includes selecting, based on the distance-to-obstacle image, a delivery point in the delivery zone. The method also includes positioning the UAV above the delivery point in the delivery zone for delivery of a payload.

Semantic adjustment of unmanned aerial vehicle delivery points

The present disclosure relates to activity monitoring systems and methods for gating whether or not steps should be counted in an observation window based on whether a decision tree concludes there are consecutive step activities (versus no activity or other activities) in the observation window. Particularly, certain aspects are directed to a method that includes obtaining acceleration data for an observation window of an accelerometer, inputting two or more characteristics of the acceleration data into a decision tree to determine activity occurring within the observation window, assigning a first class to the observation window when the determined activity is associated with consecutive steps, assigning a second class to the observation window when the determined activity is not associated with consecutive steps, and when the first class is assigned to the observation window, determining a step count for the observation window using frequency analysis.

Systems and methods for gating a step count

A computer-implemented method comprises receiving, by an image processing system, a depth image captured by a stereo camera on an unmanned aerial vehicle (UAV). One or more pixels of the depth image are associated with corresponding depth values indicative of distances of one or more objects to the stereo camera. The image processing system determines that one or more pixels of the depth image are associated with invalid depth values. The image processing system infers, based on a distribution of the one or more pixels of the depth image that are associated with invalid depth values, a presence of a potential obstacle in an environment of the UAV. The UAV is controlled based on the inferred presence of the potential obstacle.

Method for controlling an unmanned aerial vehicle to avoid obstacles

A method includes navigating, by an unmanned aerial vehicle (UAV), to a first altitude above a first delivery point at a delivery location. The method further includes determining, by the UAV, a second delivery point at the delivery location. The method includes navigating, by the UAV, through a descending trajectory to move the UAV from the first altitude above the first delivery point to a second altitude above the second delivery point at the delivery location. The second altitude is lower than the first altitude. The method additionally includes delivering, by the UAV, a payload to the second delivery point at the delivery location. The method includes after delivering the payload, navigating, by the UAV, through an ascending trajectory to move the UAV from a third altitude above the second delivery point to a fourth altitude above the first delivery point. The fourth altitude is higher than the third altitude.

Growing community of inventors

Mill Valley, CA, United States of America

Ali Shoeb