Zoox, Inc.

Nathaniel Jon Kaiser

Derek Adams

Michael Carsten Bosse

Brice Rebsamen

Elena Stephanie Stumm

Till Kroeger

Guillermo Duenas Arana

Yan Cui

Determining a localization performance metric using sensor data is described. A vehicle may capture sensor data from one or more sensors of the vehicle. The sensor data may be input into a localization component to determine localization data (e.g., a location/orientation of the vehicle) and auxiliary location data (e.g., residual data indicating a difference between the measured location/orientation and an actual or estimated location/orientation). The auxiliary localization data (which, in some examples, may include residuals associated with various sensor modalities) may be input into a machine learned model. The machine learned model may generate the localization performance metric. The vehicle may determine an action to take based on the localization performance metric. In some examples, the action may be tied to the performance meeting or exceeding a threshold.

Localization performance metric

Techniques described herein comprise receiving, from a first measurement system associated with a vehicle, first measurement data taken at a first time and second measurement data taken at a second time, different from the first time. Third measurement data taken at the first time and fourth measurement data taken at the second time is received from a second measurement system associated with the vehicle. A probability is determined that an error has occurred in the vehicle based on the first, second, third and fourth measurement data; and the vehicle is controlled based at least in part on the determined probability.

Error estimation systems and methods

Techniques for determining a location of a vehicle in an environment using sensors and determining calibration information associated with the sensors are discussed herein. A vehicle can use map data to traverse an environment. The map data can include semantic map objects such as traffic lights, lane markings, etc. The vehicle can use a sensor, such as an image sensor, to capture sensor data. Semantic map objects can be projected into the sensor data and matched with object(s) in the sensor data. Such semantic objects can be represented as a center point and covariance data. A distance or likelihood associated with the projected semantic map object and the sensed object can be optimized to determine a location of the vehicle. Sensed objects can be determined to be the same based on matching with the semantic map object. Epipolar geometry can be used to determine if sensors are capturing consistent data.

Calibration based on semantic objects

Localization based on semantic objects

Localization error handling using output is described. A computing system associated with a vehicle can receive sensor data from a sensor associated with vehicle. The computing system can determine, based at least partly on the sensor data, a first instruction for controlling the vehicle during a first period of time and a difference in pose information associated with a pose of the vehicle. Based at least partly on determining the difference, the computing system can retrieve a second instruction for controlling the vehicle during a second period of time prior to the first period of time and, based at least partly on comparing the first instruction and the second instruction, the computing system can determine whether the vehicle is to follow the first instruction or perform an alternate operation.

Localization error handling

Techniques are discussed for determining a location of a vehicle in an environment using a feature corresponding to a portion of an image representing an object in the environment which is associated with a frequently occurring object classification. For example, an image may be received and semantically segmented to associate pixels of the image with a label representing an object of an object type (e.g., extracting only those portions of the image which represent lane boundary markings). Features may then be extracted, or otherwise determined, which are limited to those portions of the image. In some examples, map data indicating a previously mapped location of a corresponding portion of the object may be used to determine a difference. The difference (or sum of differences for multiple observations) are then used to localize the vehicle with respect to the map.

Growing community of inventors

Foster City, CA, United States of America

Nathaniel Jon Kaiser