Microsoft Technology Licensing, LLC

Nvidia Corporation

Michael Kroepfl

Gur Kimchi

David Nister

John Charles Curlander

Eyal Ofek

Martin Josef Ponticelli

Ibrahim Eden

Joachim Bauer

Blaise Aguera Y Arcas

Stephen L Lawler

Amir Akbarzadeh

Birgit Henke

Hannes Hegenbarth

Yu Sheng

Ivana Stojanovic

Ishwar Kulkarni

Andrew Carley

Enliang Zheng

Michael Gruber

Scott Vernon Fynn

Konrad Karner

Franz W Leberl

Zanin Cosic

Joachim Pehserl

Stefan Bernögger

Si Lu

Jugnu Agrawal

Gerhard Neuhold

Daniel Buchmueller

Gonzalo Alberto Ramos

Julian R Walker

Roberto R Molinari

Dan Hou

Vadim Cugunovs

Tony Ernst

Neda Cvijetic

Blaise Hilary Aguera-Arcas

Frederik Schaffalitzky

Vaibhav Thukral

Ruchi Bhargava

Yonatan Wexler

Michael Grabner

Youding Zhu

Jeffrey Liu

Niharika Arora

Donald Wysocki

Jordan Marr

Embodiments of the present disclosure relate to generating RADAR (RAdio Detection And Ranging) point clouds based on RADAR data obtained from one or more RADAR sensors disposed on one or more ego-machines. In these or other embodiments, the RADAR point clouds may be used to generate map data. Additionally or alternatively, the RADAR point clouds may be used for performing localization.

Sensor data point cloud generation for map creation and localization for autonomous systems and applications

One or more embodiments of the present disclosure relate to generation of map data. In these or other embodiments, the generation of the map data may include determining whether objects indicated by the sensor data are static objects or dynamic objects. Additionally or alternatively, sensor data may be removed or included in the map data based on determinations as to whether it corresponds to static objects or dynamic objects.

Sensor data based map creation for autonomous systems and applications

One or more embodiments of the present disclosure relate to generating RADAR (RAdio Detection And Ranging) point clouds based on RADAR data obtained from one or more RADAR sensors disposed on one or more ego-machines. In these or other embodiments, the RADAR point clouds may be communicated to a distributed map system that is configured to generate map data based on the RADAR point clouds. In some embodiments of the present disclosure, certain compression operations may be performed on the RADAR point clouds to reduce the amount of data that is communicated from the ego-machines to the map system.

Sensor data based map creation and localization for autonomous systems and applications

LiDAR (light detection and ranging) and RADAR (radio detection and ranging) systems are commonly used to generate point cloud data for 3D space around vehicles, for such functions as localization, mapping, and tracking. Improved techniques for processing the point cloud data that has been collected are provided. The improved techniques include mapping one or more point cloud data points into a depth map, the one or more point cloud data points being generated using one or more sensors; determining one or more mapped point cloud data points within a bounded area of the depth map, and detecting, using one or more processing units and for an environment surrounding a machine corresponding to the one or more sensors, a location of one or more entities based on the one or more mapped point cloud data points.

Analysis of point cloud data using depth maps

An end-to-end system for data generation, map creation using the generated data, and localization to the created map is disclosed. Mapstreams&#x2014;or streams of sensor data, perception outputs from deep neural networks (DNNs), and/or relative trajectory data&#x2014;corresponding to any number of drives by any number of vehicles may be generated and uploaded to the cloud. The mapstreams may be used to generate map data&#x2014;and ultimately a fused high definition (HD) map&#x2014;that represents data generated over a plurality of drives. When localizing to the fused HD map, individual localization results may be generated based on comparisons of real-time data from a sensor modality to map data corresponding to the same sensor modality. This process may be repeated for any number of sensor modalities and the results may be fused together to determine a final fused localization result.

Map creation and localization for autonomous driving applications

A system that facilitates collecting data is described herein. The system includes a digital camera that is configured to capture images in a visible light spectrum and a near-infrared camera that is configured to capture near infrared images, wherein a field of view of the digital camera and the field of view of the near-infrared camera are substantially similar. The system further includes a trigger component that is configured to cause the digital camera and the near-infrared camera to capture images at a substantially similar point in time, and also includes a mounting mechanism that facilitates mounting the digital camera and the near-infrared camera to an automobile.

Three dimensional image capture system for imaging building facades using a digital camera, a near-infrared camera, and laser range finder

Various types of systems or technologies can be used to collect data in a 3D space. For example, LiDAR (light detection and ranging) and RADAR (radio detection and ranging) systems are commonly used to generate point cloud data for 3D space around vehicles, for such functions as localization, mapping, and tracking. This disclosure provides improved techniques for processing the point cloud data that has been collected. The improved techniques include mapping 3D point cloud data points into a 2D depth map, fetching a group of the mapped 3D point cloud data points that are within a bounded window of the 2D depth map; and generating geometric space parameters based on the group of the mapped 3D point cloud data points. The generated geometric space parameters may be used for object motion, obstacle detection, freespace detection, and/or landmark detection for an area surrounding a vehicle.

Various types of systems or technologies can be used to collect data in a 3D space. For example, LiDAR (light detection and ranging) and RADAR (radio detection and ranging) systems are commonly used to generate point cloud data for 3D space around vehicles, for such functions as localization, mapping, and tracking. This disclosure provides improvements for processing the point cloud data that has been collected. The processing improvements include analyzing point cloud data using trajectory equations, depth maps, and texture maps. The processing improvements also include representing the point cloud data by a two dimensional depth map or a texture map and using the depth map or texture map to provide object motion, obstacle detection, freespace detection, and landmark detection for an area surrounding a vehicle.

Analysis of point cloud data using depth and texture maps

Various types of systems or technologies can be used to collect data in a 3D space. For example, LiDAR (light detection and ranging) and RADAR (radio detection and ranging) systems are commonly used to generate point cloud data for 3D space around vehicles, for such functions as localization, mapping, and tracking. This disclosure provides improvements for processing the point cloud data that has been collected. The processing improvements include using a three dimensional polar depth map to assist in performing nearest neighbor analysis on point cloud data for object detection, trajectory detection, freespace detection, obstacle detection, landmark detection, and providing other geometric space parameters.

Analysis of point cloud data using polar depth maps and planarization techniques

Vehicle-mounted sensor system that includes cameras and laser measurement systems

Three dimensional image capture system for imaging building facades using a digital camera, near-infrared camera, and laser range finder

An image matching system is described for finding reference images that match a query image. The image matching system performs this operation in expeditious fashion by restricting the matching to a bounding region; the bounding region, in turn, may be associated with a location at which the query image may have been captured. In addition, the image matching system provides various mechanisms that expedite adding new reference images to an image index, to thereby provide a re-enforced learning mechanism of the image matching system.

Spatial image index and associated updating functionality

A system for displaying hybrid image data produced by embedding additional media objects within street-level panoramic images includes a user interface through which a user may view, search for, and/or navigate through additional media objects in the context of browsing a virtual environment of a location at street level. In response to user input indicating a request to view a geographic location and/or an additional media object, street-level panoramic image data associated with the geographic location, in which one or more additional media objects also associated with the geographic location have been embedded, may be provided for display through the user interface. The user interface may be provided by a client device including one or more processors that receive hybrid image data produced by one or more processors of a server and display the image data to the user.

Viewing media in the context of street-level images

A digital camera described herein includes an analyzer component that analyzes a histogram of a first image, wherein the first image has a first resolution. The digital camera also includes a setter component that sets an exposure time for capturing a second image based at least in part upon the analysis of the histogram of the first image, wherein the second image has a second resolution, and wherein the first resolution of the first image is less than the second resolution of the second image.

Determining exposure time in a digital camera

Photographs of an object may be oriented with respect to both the geographic location and orientation of the object by registering a 3D model derived from a plurality of photographs of the objects with a 2D image of the object having a known location and orientation. For example, a 3D point cloud of an object created from photographs of the object using a Photosynth&#x2122; tool may be aligned with a satellite photograph of the object, where the satellite photograph has location and orientation information. A tool providing scaling and rotation of the 3D model with respect to the 2D image may be used or an automatic alignment may be performed using a function based on object edges filtered at particular angles. Once aligned, data may be recorded that registers camera locations for the plurality of photographs with geographic coordinates of the object, either absolute latitude/longitude or relative to the object.

Spatially registering user photographs

Growing community of inventors

Redmond, WA, United States of America

Michael Kroepfl