Zoox, Inc.

Till Kroeger

Michael Carsten Bosse

Elena Stephanie Stumm

Guillermo Duenas Arana

Nathaniel Jon Kaiser

Soroush Dean Khadem

Ryan McMichael

Derek Adams

Johannes Edren

Robert Nicholas Moor

Taylor Andrew Arnicar

Veeresh Taranalli

Brice Rebsamen

Maxwell Yaron

An evaluation computing system may implement techniques to validate a vehicle controller, such as based on a detection of a systematic fault. The evaluation computing system may access data (e.g., log data and/or map data) associated with an operation of the vehicle in an environment as controlled by the controller. The evaluation computing system may modify a portion of the map data representative of a simulated change associated with a portion of the environment. The evaluation computing system may run a simulation with a simulated environment, generated based on the modified map data, to determine whether the controller detects and/or mitigates the simulated change in a sufficient manner. Based on a determination of whether or not the controller detects and/or mitigates the simulated change in a sufficient manner, the evaluation computing system may determine an error associated with the controller or may validate the controller.

Systematic fault detection in vehicle control systems

Techniques for estimating vehicle poses are described herein. A pose may refer to the inertial pose or position of a vehicle. The pose of the vehicle may be used to determine a position of the vehicle in an environment, for determining a trajectory of the vehicle through the environment, or the like. In examples, a pose determination component of an autonomous vehicle can determine a pose at a frequency that is different from other systems on the vehicle and/or that is not synchronized with those systems. The techniques described herein allow for estimating poses at any measurement time, e.g., to determine a pose at a measurement time at which other data is measured, generated, or the like.

Data estimation in asynchronous systems

Techniques for determining a location of a vehicle in an environment using sensors and determining calibration information associated with the sensors are discussed herein. A vehicle can use map data to traverse an environment. The map data can include semantic map objects such as traffic lights, lane markings, etc. The vehicle can use a sensor, such as an image sensor, to capture sensor data. Semantic map objects can be projected into the sensor data and matched with object(s) in the sensor data. Such semantic objects can be represented as a center point and covariance data. A distance or likelihood associated with the projected semantic map object and the sensed object can be optimized to determine a location of the vehicle. Sensed objects can be determined to be the same based on matching with the semantic map object. Epipolar geometry can be used to determine if sensors are capturing consistent data.

Calibration based on semantic objects

A soft-constraint technique for refining an initial pose graph may eschew using a hard constraint that identifies different sensor data and/or poses as necessarily being associated with a same portion of an environment. Instead, the soft-constraint technique may employ a loss function with a convergence basin that may be defined based at least in part on an object classification that strongly penalizes candidate locations within a distance associated with the convergence basin. These candidate locations may be based at least in part on one or more object detections associated (1:1) with one or more poses of the initial pose graph. This may result in one or more candidate locations that do not merge with other candidate locations, giving the pose graph optimization the permissiveness or softness according to the techniques described herein.

Secondary sensor data-based soft constraint optimization for pose graph and mapping refinement

Techniques for determining map data for localizing a vehicle in an environment are discussed herein. A vehicle can capture sensor data such as image data and can detect objects in the sensor data. Detected objects can include semantic object such as traffic lights, lane lines, street signs, poles, and the like. Rays based on a pose of the sensor and the detected objects can be determined. Intersections between the rays can be determined. The intersections can be utilized to determine candidate landmarks in the environment. The candidate landmarks can be utilized to determine landmarks and to eliminate other candidate landmarks. The map data can be determined based on the three-dimensional locations and utilized to localize vehicles.

Determining data for semantic localization

Motion can be induced at a vehicle, e.g., by actuating components of an active suspension system, and first sensor data and second sensor data representing an environment of the vehicle can be captured at a first position and a second position, respectively, resulting from the induced motion. A second sensor can determine motion information associated with the first position and the second position. Calibration information about the sensor, the first sensor data, and the motion information can be used to determine an expectation of sensor data at the second position. A calibration error can be the difference between the second sensor data and the expected sensor data.

Sensor calibration and verification using induced motion

This disclosure is directed to calibrating sensors mounted on an autonomous vehicle. A dense depth map can be generated in a two-dimensional camera space using point cloud data generated by one of the sensors. Image data from another of the sensors can be compared to the dense depth map in the two-dimensional camera space. Differences determined by the comparison can indicate alignment errors between the sensors. Calibration data associated with the errors can be determined and used to calibrate the sensors without the need for calibration infrastructure.

Sensor calibration using dense depth maps

Localization based on semantic objects

This disclosure is directed to validating a calibration of and/or calibrating sensors using semantic segmentation information about an environment. For example, the semantic segmentation information can identify bounds of objects, such as invariant objects, in the environment. Techniques described herein may determine sensor data associated with the invariant objects and compare that data to a feature known from the invariant object. Misalignment of sensor data with the known feature can be indicative of a calibration error. In some implementations, the calibration error can be determined as a distance between the sensor data and a line or plane representing a portion of the invariant object.

Sensor validation using semantic segmentation information

This disclosure is directed to calibrating sensors mounted on an autonomous vehicle. First image data and second image data representing an environment can be captured by first and second cameras, respectively (and or a single camera at different points in time). Point pairs comprising a first point in the first image data and a second point in the second image data can be determined and projection errors associated with the points can be determined. A subset of point pairs can be determined, e.g., by excluding point pairs with the highest projection error. Calibration data associated with the subset of points can be determined and used to calibrate the cameras without the need for calibration infrastructure.

Sensor calibration

This disclosure is directed to calibrating sensor arrays, including sensors arrays mounted on an autonomous vehicle. Image data from multiple cameras in the sensor array can be projected into other camera spaces using one or more dense depth maps. The dense depth map(s) can be generated from point cloud data generated by one of the sensors in the array. Differences determined by the comparison can indicate alignment errors between the cameras. Calibration data associated with the errors can be determined and used to calibrate the sensor array without the need for calibration infrastructure.

Camera calibration using dense depth maps

Techniques for calibrating a camera utilizing a calibration station. The camera calibration station may be configured to emit collimated light toward a camera housed in a cradle. The camera may capture images of the collimated light at pre-determined positions throughout a calibration sequence. A computing system associated with the camera calibration station may utilize reference locations determined based on the collimated light at the pre-determined positions compared to measured locations of the collimated light at the pre-determined positions to determine intrinsics of the camera (e.g., focal length of lens, optical center of lens, etc.) and an error associated therewith. Based at least in part on the error being less than a threshold error, the computing system may store the intrinsic parameters of the camera.

Growing community of inventors

San Francisco, CA, United States of America

Till Kroeger