Zoox, Inc.

Michael Carsten Bosse

Derek Adams

Brice Rebsamen

Francesco Papi

Elena Stephanie Stumm

Patrick Blaes

Veeresh Taranalli

Subhasis Das

Till Kroeger

Zachary Sun

Chuang Wang

Nathaniel Jon Kaiser

Shiwei Sheng

Aleksandrs Ecins

Gerry Chen

Siddharth Ratapani Navin

Venkata Subrahmanyam Chandra Sekhar Chebiyyam

Subasingha Shaminda Subasingha

Sarah Tariq

Johannes Edren

David Burdick Berman

Joshua Kriser Cohen

Nam Gook Cho

Taylor Andrew Arnicar

Guillermo Duenas Arana

Amanda Brown Prescott

Jonathan Tyler Dowdall

Adrian Michael Costantino

Jacob Daniel Boydston

Jifei Qian

Eric Andre Senant

Shuangting Liu

Yassen Ivanchev Dobrev

John Bryan Carter

Anton Mario Bongio Karrman

Nicholas George Dilip Roy

Dilip Bethanabhotla

Anne-Claire Elisabeth Marie Le Hénaff

Weihung Ko

Qi Fu

Hirotatsu Armstrong

Jeffrey Eric Tolliver

Feroze Naina Mohamed Dheen Mohamed Ismail

Techniques are described for modifying a factor associated with an invalid point cloud registration (e.g., alignment) between two data sets, to improve a map generation process. A computing system receives sensor data from a vehicle traversing an environment. The computing system aligns discrete data sets, such as to generate a representation of the environment based on the discrete data sets. In some examples, the computing system can apply an optimization algorithm to the alignments, to optimize the representation of the environment. The computing system identifies an invalid (non-ideal) alignment between two data sets and modifies a factor associated with the invalid alignment, introducing a repulsive force. The computing system performs an optimization with the modified factor, which causes a location associated with at least one of the two data sets to be different from a previously measured location. The computing system generates a map based on the optimization.

Aligning data for mapping

Techniques for integrating sensor data into a scene or map based on statistical data of captured environmental data are discussed herein. The data may be stored as a multi-resolution voxel space and the techniques may comprise first applying a pre-alignment or localization technique prior to fully integrating the sensor data.

System for aligning sensor data with maps comprising covariances

Systems and techniques for determining more accurate object parameter values, such as orientation values (e.g., yaw, location, position, etc.), for objects detected in an environment are disclosed. A vehicle computing system may receive sensor data from multiple sensor systems that indicates a parameter value determined at the individual sensor systems. The vehicle computing system may determine values and probabilities for particular parameters modes based on the sensor data parameter values and may further filter these values using a mixture model to determine probability distributions for the modes and associated values. These filtered values and modes may then be used to determine predicted object trajectories that can be used to control a vehicle.

Determining object orientation based on parameter modes

A computer-implemented method. Includes determining an estimate of a state of an object detected at a first time step, based on a pointwise estimate of the state of the object at the first time step and pointwise measurements of the state of the object at a plurality of further time steps. Includes generating, using the estimate of the state of the object, a proposed annotation associated with the object at the first time step. Includes rendering, via a user interface, a visual representation of the environment at the first time step and a visual representation of the proposed annotation. Includes receiving, via the user interface, user input indicating a user-approved annotation associated with the object at the first time step. Includes generating training data for a machine learning model for use in controlling an autonomous vehicle, based at least in part on the user-approved annotation.

Assisted labelling of training data for machine learning models

Relocating and/or re-sizing map elements using an updated pose graph without introducing abnormalities to the map data may comprise determining a transformation between a source node of a first pose graph and a target node of a second pose graph and determining a modification to a map element based at least in part on the transformation. The techniques may include determining a stress on the map based at least in part on one or more modifications to map elements and determining if the stress meets or exceeds a threshold. In instances where the stress meets or exceeds a threshold, a modification may be altered, reversed, and/or indicated in a notification transmitted to a user interface.

Map annotation modification using sparse pose graph node updates

Techniques for representing a scene or map based on statistical data of captured environmental data are discussed herein. In some cases, the data (such as covariance data, mean data, or the like) may be stored as a multi-resolution voxel space that includes a plurality of semantic layers. In some instances, individual semantic layers may include multiple voxel grids having differing resolutions. Multiple multi-resolution voxel spaces may be merged to generate combined scenes based on detected voxel covariances at one or more resolutions.

Using maps comprising covariances in multi-resolution voxels

Techniques for representing a scene or map based on statistical data of captured environmental data are discussed herein. In some cases, the data (such as covariance data, mean data, or the like) may be stored as a multi-resolution voxel space that includes a plurality of semantic layers. In some instances, individual semantic layers may include multiple voxel grids having differing resolutions. Multiple multi-resolution voxel spaces may be merged or aligned to generate combined scenes based on detected voxel covariances at one or more resolutions.

System and method for generating multi-resolution voxel spaces

Techniques for estimating vehicle poses are described herein. A pose may refer to the inertial pose or position of a vehicle. The pose of the vehicle may be used to determine a position of the vehicle in an environment, for determining a trajectory of the vehicle through the environment, or the like. In examples, a pose determination component of an autonomous vehicle can determine a pose at a frequency that is different from other systems on the vehicle and/or that is not synchronized with those systems. The techniques described herein allow for estimating poses at any measurement time, e.g., to determine a pose at a measurement time at which other data is measured, generated, or the like.

Data estimation in asynchronous systems

Techniques for determining errors or drifts between maps used for updating maps and/or controlling a system which uses the map. In some examples, a first, global, map may be received or determined. Sensor data may then be used to localize a system with respect to the first map and to generate a first trajectory relative to the first map. The sensor data may be used to create a second map and a second trajectory for navigating the system relative to the second map. Differences between the first and second trajectories (or portions thereof), when compared in a common reference frame, may be used as an indication of drift between various processes or errors in the maps and, subsequently, be used for updating the first map and/or controlling the system.

Drift detection

Techniques associated with generating and maintaining sparse geographic and map data. In some cases, the system may maintain a factor graph comprising a plurality of nodes. In some cases, the nodes may comprise pose data and sensor data associated with an autonomous vehicle at the geographic position represented by the node. The nodes may be linked based on shared trajectories and shared sensor data.

System for sparsely representing and storing geographic and map data

Systems and methods for calibrating multiple inertial measurement units on a system include calibrating a first of the inertial measurement units relative to the system using a first calibration model, and calibrating the remaining inertial measurement unit(s) relative to the first inertial measurement unit using a second calibration model. The calibration of the remaining inertial measurement unit(s) to the first inertial measurement unit can be based on a rigid body model by aligning a rotational velocity of the first inertial measurement unit with a rotational velocity of the remaining inertial measurement unit(s).

Calibrating multiple inertial measurement units

A soft-constraint technique for refining an initial pose graph may eschew using a hard constraint that identifies different sensor data and/or poses as necessarily being associated with a same portion of an environment. Instead, the soft-constraint technique may employ a loss function with a convergence basin that may be defined based at least in part on an object classification that strongly penalizes candidate locations within a distance associated with the convergence basin. These candidate locations may be based at least in part on one or more object detections associated (1:1) with one or more poses of the initial pose graph. This may result in one or more candidate locations that do not merge with other candidate locations, giving the pose graph optimization the permissiveness or softness according to the techniques described herein.

Secondary sensor data-based soft constraint optimization for pose graph and mapping refinement

Techniques described are related to determining when a discrepancy between data of multiple sensors (e.g., IMUs) might be attributable to a sensor error, as opposed to operating conditions, such as sensor bias or noise. For example, the sensor data is passed through one or more filters (e.g., bandpass filter) that model the bias or noise, and the filtered data may then be compared for consistency. In some examples, consistency may be based on residuals or some other metric describing discrepancy among the filtered sensor data.

Sensor consensus monitor

Techniques for using a set of variables to estimate a vehicle velocity of a vehicle are discussed herein. A system may determine an estimated velocity of the vehicle using a minimization based on an initial estimated velocity, steering angle data and wheel speed data. The system may then control an operation of the vehicle based at least in part on the estimated velocity.

Estimating vehicle velocity

Growing community of inventors

Cupertino, CA, United States of America

Michael Carsten Bosse