Zoox, Inc.

Light Labs Inc.

Nitesh Shroff

Jesse Sol Levinson

Rajiv Laroia

Brice Rebsamen

Sapna A Shroff

Elena Stephanie Stumm

Derek Adams

Ashutosh Gajanan Rege

Zeng Wang

James William Vaisey Philbin

Kari Pulli

David Pfeiffer

Dragomir Dimitrov Anguelov

Subhasis Das

Harpuneet Singh

Aleksandrs Ecins

Nagilla Dikpal Reddy

John Sasinowski

Weiguang Si

Nghi Phan

Jiamin Bai

A method includes receiving a first signal from a first sensor, the first signal including data representing an environment. The method also includes receiving a second signal from a second sensor, the second signal including data representing the environment. The method further includes determining a group of objects based at least in part on the received data, and identifying an error associated with data included in the first signal and/or the second signal.

Detecting errors in sensor data

Techniques are discussed for using multi-resolution maps, for example, for localizing a vehicle. Map data of an environment can be represented by discrete map tiles. In some cases, a set of map tiles can be precomputed as contributing to localizing the vehicle in the environment, and accordingly, the set of map tiles can be loaded into memory when the vehicle is at a particular location in the environment. Further, a level of detail represented by the map tiles can be based at least in part on a distance between a location associated with the vehicle and a location associated with a respective region in the environment. The level of detail can also be based on a speed of the vehicle in the environment. The vehicle can determine its location in the environment based on the map tiles and/or the vehicle can generate a trajectory based on the map tiles.

Multi-resolution maps for localization

Techniques are discussed for segmenting sensor data captured by a sensor to remove a ground surface from the sensor data. A first technique includes capturing sensor data represented as multichannel image and segmenting the image according to image processing techniques. The ground surface can be removed from the sensor data, and a subset of the sensor data can be associated with a voxel space. A second technique includes capturing sensor data and unprojecting the sensor data to generate three dimensional data, which can be associated with a voxel space. Ground plane data associated with a location can be accessed or determined and voxel data that is within a threshold height of the ground plane data can be removed from the voxel space. Clustering techniques can determine objects represented in the data, and a vehicle can be controlled based on the objects.

Time of flight data segmentation

Techniques for decimating portions of a map of an environment are discussed herein. The environment can be represented by a three-dimensional (3D) map including a plurality of polygons and semantic information associated with the polygons. In some cases, decimation operations may be based on semantic information associated with the environment. Differing decimation operations and/or levels may be applied to polygons of different semantic classifications or differing contribution levels. Boundaries between regions having different semantic information can be preserved. Meshes can be decimated using different decimation operators or decimation levels and an accuracy of localizing can be compared using the various decimated meshes. An optimal mesh can be selected and sent to vehicles for localizing the vehicles in the environment.

Mesh decimation based on semantic information

A system may receive a sensor dataset representing an environment and use the dataset to create or update a map. In creating or updating the map, the system may determine an object classification of one or more detected objects and only selectively incorporate data into the map based at least in part on the classification. The map may be associated with the classification (or semantic) information of the objects, as well as weights based on the classification. Similarly, datasets with selected classes of data removed may be used for system localization. Further, the system may determine an object track of the objects. When updating the map, voxels in a voxel space may indicate an occupied voxel based on a threshold number of observances. The object track and clean map can then be used for controlling an autonomous vehicle.

Creating clean maps including semantic information

Techniques for validating data that represents an environment are discussed herein. In examples, the techniques can associate data of an environment with a voxel space including multiple voxels. The techniques can analyze the voxels to identify an error, such as multiple layers for a single surface in an environment, a hole for a location in an environment where a hole does not exist, uneven regions for a smooth surface in an environment, and so on. In some examples, when an error has been identified, an action can be performed, such as displaying a representation associated with the data and an indication of the error, storing an indication of the error in a log file, updating a trajectory for a vehicle, updating the data or a mesh associated with the data, and so on.

Mesh validation

Loading multi-resolution maps for localization

Mesh decimation techniques and validation

Techniques for generating maps without shadows are discussed herein. A plurality of images can be captured by a vehicle traversing an environment representing various perspectives and/or lighting conditions in the environment. A shadow within an image can be identified by a machine learning algorithm trained to detect shadows in images and/or by projecting the image onto a three-dimensional (3D) map of the environment and identifying candidate shadow regions based on the geometry of the 3D map and the location of the light source. Shadows can be removed or minimized by utilizing blending or duplicating techniques. Color information and reflectance information can be added to the 3D map to generate a textured 3D map. A textured 3D map without shadows can be used to simulate the environment under different lighting conditions.

Generating maps without shadows

Generating maps without shadows using geometry

In various embodiments a camera with multiple optical chains, e.g., camera modules, is controlled to operate in one of a variety of supported modes of operation. The modes include a non-motion mode, a motion mode, a normal burst mode and/or a reduced data burst mode. Motion mode is well suited for capturing an image including motion, e.g., moving object(s) with some modules being used to capture scene areas using a shorter exposure time than other modules and the captured images then being combined taking into consideration locations of motion. A reduced data burst mode is supported in some embodiments in which camera modules with different focal lengths capture images at different rates. While the camera modules of different focal length operate at different image capture rates in the reduced data burst mode, images are combined to support a desired composite image output rate, e.g., a desired frame rate.

Methods and apparatus for supporting burst modes of camera operation

An exemplary camera calibration apparatus includes a movable, e.g., rotatable, support structure which is controllably positioned to allow for image capture of different test patterns and image capture of the same pattern at different distances by a mounted camera. A first test pattern is mounted on a wall, e.g., a pyramid shaped 4 sided wall formed by panels surrounding the camera under calibration. The movable support structure has a first mirror attached to a first side and has a second test pattern attached to a second side. A second mirror mounted on an internal sidewall of the calibration apparatus housing facilities a different image path distance between the camera capturing the image of the first test pattern and the first test pattern. The exemplary camera calibration apparatus is well suited for efficiently calibrating camera devices including a plurality of camera modules, e.g., optical chains, in a relatively small area.

Growing community of inventors

Millbrae, CA, United States of America

Nitesh Shroff