Amazon Technologies, Inc.

Waymo LLC

Samsung Electronics Co., Ltd.

Volodymyr V Ivanchenko

Kenneth Mark Karakotsios

Isaac Scott Noble

Colin Andrew Braley

Volker Grabe

Kah Kuen Fu

Leo Benedict Baldwin

Sharadh Ramaswamy

Dong Zhou

Gunnar Hovden

Mingjing Huang

Alexander Meade

Jeffrey P Bezos

Daniel Bibireata

Geoffrey Heller

Richard Howard Suplee, Iii

Gregory Michael Hart

Yu Zhang

Howard Look

David Wayne Stafford

Steven Ka Cheung Moy

Gil Shotan

Dennis R Hodge

Example embodiments relate to identification of proxy calibration targets for a fleet of sensors. An example method includes collecting, using a sensor coupled to a vehicle, data about one or more objects within an environment of the vehicle. The sensor has been calibrated using a ground-truth calibration target. The method also includes identifying, based on the collected data, at least one candidate object, from among the one or more objects, to be used as a proxy calibration target for other sensors coupled to vehicles within a fleet of vehicles. Further, the method includes providing, by the vehicle, data about the candidate object for use by one or more vehicles within the fleet of vehicles.

Identification of proxy calibration targets for a fleet of vehicles

A method includes monitoring, at a computing device, outputs of a sensor validator. Each output is generated by the sensor validator based on corresponding sensor data from a sensor coupled to an autonomous vehicle, and each output indicates whether the corresponding sensor data is associated with an event. The method also includes mutating, at the computing device, particular sensor data to generate mutated sensor data that is associated with a particular event. The method further includes determining, at the computing device, a performance metric associated with the sensor validator based on a particular output generated by the sensor validator. The particular output is based on the mutated sensor data.

Methods and apparatus for monitoring a sensor validator

In one example, a method is provided that includes receiving lidar data obtained by a lidar device. The lidar data includes a plurality of data points indicative of locations of reflections from an environment of the vehicle. The method includes receiving images of portions of the environment captured by a camera at different times. The method also includes determining locations in the images that correspond to a data point of the plurality of data points. Additionally, the method includes determining feature descriptors for the locations of the images and comparing the feature descriptors to determine that sensor data associated with at least one of the lidar device, the camera, or a pose sensor is accurate or inaccurate.

Methods and apparatus for validating sensor data

Example embodiments relate to methods for localizing light detection and ranging (lidar) calibration targets. An example method includes generating a point cloud of a region based on data from a light detection and ranging (lidar) device. The point cloud may include points representing at least a portion of a calibration target. The method also includes determining a presumed location of the calibration target. Further, the method includes identifying, within the point cloud, a location of a first edge of the calibration target. In addition, the method includes performing a comparison between the identified location of the first edge of the calibration target and a hypothetical location of the first edge of the calibration target within the point cloud if the calibration target were positioned at the presumed location. Still further, the method includes revising the presumed location of the calibration target based on at least the comparison.

Methods for localizing light detection and ranging (lidar) calibration targets

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for determining an alignment between cross-modal sensor data. In one aspect, a method comprises: obtaining (i) an image that characterizes a visual appearance of an environment, and (ii) a point cloud comprising a collection of data points that characterizes a three-dimensional geometry of the environment; processing each of a plurality of regions of the image using a visual embedding neural network to generate a respective embedding of each of the image regions; processing each of a plurality of regions of the point cloud using a shape embedding neural network to generate a respective embedding of each of the point cloud regions; and identifying a plurality of region pairs using the embeddings of the image regions and the embeddings of the point cloud regions.

Cross-modal sensor data alignment

A user can provide input to an electronic device by performing a specific motion or gesture that can be detected by the device. At least one imaging or detection element captures information including the motion or gesture, such that one or more dwell points can be determined in two or three dimensions of space. The dwell points can correspond to any point where the motion pauses for at least a minimum amount of time, such as at an endpoint or a point where the motion significantly changes or reverses direction. The set of dwell points, and the order in which those dwell points occur, can be compared against a set of gestures to attempt to match a gesture associated with a particular input. Such an approach is useful for devices with image capture elements or other components that are not able to accurately capture motion or determine movements, etc.

Overcoming motion effects in gesture recognition

Image capture can be improved by capturing a sequence of images and analyzing the images to select the image with the least blur and/or an acceptable amount of blur. Gradients can be calculated for at least a portion of the images, and gradient histograms generated. Two or more component curves can be fit to each histogram, such as by using a Gaussian mixture model, and the curves can be compared to determine an amount of variation between the curves. The image with the smallest differences between component curves, or with differences less than a specified blur threshold, can be selected as a sufficiently sharp image and provided for viewing, processing, or another intended purpose of the image to be captured.

Dynamic image selection

A user attempting to obtain information about an object can capture image information including a view of that object, and the image information can be used with a matching or identification process to provide information about that type of object to the user. Information about the orientation of the camera and/or device used to capture the image can be provided in order to limit an initial search space for the matching or identification process. In some embodiments, images can be selected for matching based at least in part upon having a view matching the orientation of the camera or device. In other embodiments, images of objects corresponding to the orientation can be selected. Such a process can increase the average speed and efficiency in locating matching images. If a match cannot be found in the initial space, images of other views and categories can be analyzed as well.

Orientation-assisted object recognition

An apparatus and method for reducing noise in a depth image are provided. The method includes capturing an image, generating a depth map of the image, the depth map including depth information corresponding to at least one object included in the image, performing over-segmentation of the depth map to generate segmentation information corresponding to the image, determining at least one segment to be noise according to the over-segmentation, and deleting the segment determined to be noise.

Apparatus and method for noise reduction in depth images during object segmentation

The registration of images captured at multiple locations, for purposes such as location mapping, can be improved by utilizing multiple image capture elements pointing in at least two different directions or having different viewable ranges. At least one primary image is captured at each location. If the primary image is not able to be correlated with at least one other image based on one or more matching features, image information captured by at least one other camera at substantially the same times as those images can be analyzed to attempt to determine a change in position and/or orientation of the device between those times, which can assist in correlating the primary images. In some embodiments, motion or orientation determining elements can assist in the determination of device movement, and in at least some embodiments can reduce the amount of image information to be processed for a match.

Registration of low contrast images

The quality of stereoscopic imaging using cameras with wide angle lenses can be improved using various calibration approaches to determine distortions or misalignments between the cameras. A calibration object such as a checkered grid with a determined inclination between portions can be used to provide lateral calibration as well as depth information. Intersections of the checkered regions can be used to determine intersection points for the calibration object in the image. Lines formed by the checkered regions can be located, and the points correlated with these lines, in order to determine correspondence between points in the image. These points are mapped to corresponding points of the calibration object in the real world, to determine system parameters and/or image adjustments to be made when subsequent images are captured, in order to remove the distortions and misalignment when providing a stereo image based on images from the cameras.

Approaches for rectifying stereo cameras

The quality of stereoscopic imaging using cameras with wide angle lenses can be improved using various calibration approaches to determine distortions or misalignments between the cameras. A calibration object such as a checkered grid with a determined inclination between portions can be used to provide lateral calibration as well as depth information. Intersections of the checkered regions can be used to determine intersection points for the calibration object in the image. Regions about at least a portion of these points can be analyzed, where the regions are sized such that features of the calibration object are substantially linear. Nearest neighbor points can be identified, and a local orientation determined in order to determine a relationship of each of the nearest points. Once identified, the points in each image can be correlated to corresponding points of the calibration object.

Growing community of inventors

Mountain View, CA, United States of America

Volodymyr V Ivanchenko