Zoox, Inc.

Subhasis Das

Zeng Wang

Dragomir Dimitrov Anguelov

Francesco Papi

Chuang Wang

Derek Xiang Ma

Bertrand Robert Douillard

Benjamin Isaac Zwiebel

Michael Carsten Bosse

Adrian Michael Costantino

Shida Shen

Jesse Sol Levinson

David Pfeiffer

Allan Zelener

Shiwei Sheng

Carden Taylor Bagwell

Xinyu Xu

Kai Yu

Philippe Martin Burlina

Siavosh Rezvan Behbahani

Andrew Scott Crego

Nam Gook Cho

Scott M Purdy

Mark Alan Bates

Qian Song

Oytun Ulutan

Sai Anurag Modalavalasa

Sabeek Mani Pradhan

Zachary Sun

Antonio Prioletti

Jifei Qian

Yassen Ivanchev Dobrev

Aditya Pramod Khadilkar

Po-Jen Lai

Minsu Jang

Yi-Ting Lin

Cheng-Hsin Wuu

Gerry Chen

Troy Donovan O'Neal

Subasingha Shaminda Subasingha

James William Vaisey Philbin

Markus Jost

Nitesh Shroff

Brice Rebsamen

Yuanyuan Chen

Mark Jonathon McClelland

Shaminda Subasingha

Amanda Brown Prescott

Zejia Zheng

Xuan Zhong

Amir Ghaderi

He Yi

Anton Mario Bongio Karrman

Jeffrey Ronald Pyke

Dhanushka Nirmevan Kularatne

Marina Camille Josephs

Peng Wang

Liujiang Yan

Valerie Bumbaca Randolph

Ruijie He

Troy Donovan O′Neal

Venkata Subrahrnanyarn Chandra Sekhar Chebiyyam

Techniques for determining a presence of an object, especially an object such as animal or debris, in a path of a vehicle, are discussed herein. For example, sensors of various modalities, which may include multispectral sensors, may capture data representing an environment the vehicle is traversing. In examples, one or more trained machine learned (ML) models, operating on a vehicle computing system, may detect and/or classify objects in the environment, based on input data of one or more modalities or spectral bands. The ML models may be pre-trained using training data including real sensor data, synthetic data, and/or augmented data, along with auto-generated annotations. In some examples, hyperspectral data may be used to identify materials associated with detected objects. A confidence score associated with the detection of the object may also be computed. The vehicle may be controlled based on detection of the object and its classification.

Training of multi-modality object detectors

Techniques are discussed herein for controlling autonomous vehicles within a driving environment, including generating and using bounding contours associated with objects detected in the environment. Image data may be captured and analyzed to identify and/or classify objects within the environment. Image-based and/or lidar-based techniques may be used to determine depth data associated with the objects, and a bounding contour may be determined based on the object boundaries and associated depth data. An autonomous vehicle may use the bounding contours of objects within the environment to classify the objects, predict the positions, poses, and trajectories of the objects, and determine trajectories and perform other vehicle control actions while safely navigating the environment.

Object bounding contours based on image data

Techniques for detecting and labeling traffic lights and/or traffic signs and the traffic lanes with which such lights and signs are associated in an environment are disclosed. Images may be evaluated to identify pixels that may be associated with a light, sign, or lane. Associations between lights and/or signs and lanes in the environment may be determined along with probabilities for the individual pixels that the pixels may be associated with one of the light/sign and lane associations. Those pixels having a sufficient probability of be associated with a light/sign and lane association may be assigned a corresponding label. An output image with such labels can be provided for vehicle control and for other operations, such as top-down segmentation and trajectory determination.

Systems and methods for traffic light detection

Techniques for updating data operations in a perception system are discussed herein. A vehicle may use a perception system to capture data about an environment proximate to the vehicle. The perception system may receive image data, lidar data, and/or radar data to determine information about an object in the environment. As different sensors may be associated with different time periods for capturing and/or processing operations, the techniques include updating object data with data from sensors associated with a shorter time period to generate intermediate object data. Such intermediate object data may reduce a delay in updating a position of an object in an environment.

Out of sequence perception system

Object detection using multispectral data

A vehicle computing system may implement techniques to determine whether two objects in an environment are related as an articulated object. The techniques may include applying heuristics and algorithms to object representations (e.g., bounding boxes) to determine whether two objects are related as a single object with two portions that articulate relative to each other. The techniques may include predicting future states of the articulated object in the environment. One or more model(s) may be used to determine presence of the articulated object and/or predict motion of the articulated object in the future. Based on the presence and/or motion of the articulated object, the vehicle computing system may control operation of the vehicle.

Articulated object determination

A computer-implemented method. Includes determining an estimate of a state of an object detected at a first time step, based on a pointwise estimate of the state of the object at the first time step and pointwise measurements of the state of the object at a plurality of further time steps. Includes generating, using the estimate of the state of the object, a proposed annotation associated with the object at the first time step. Includes rendering, via a user interface, a visual representation of the environment at the first time step and a visual representation of the proposed annotation. Includes receiving, via the user interface, user input indicating a user-approved annotation associated with the object at the first time step. Includes generating training data for a machine learning model for use in controlling an autonomous vehicle, based at least in part on the user-approved annotation.

Assisted labelling of training data for machine learning models

Techniques for using an inference wrapper to execute a machine learned model using various types of machine learning engines are disclosed. A resource allocator may interoperate with the inference wrapper to request the utilization of resources, such as memory, and generate and log utilization data. The utilization data can be used to generate visual representations of resource utilization that can then be used to improve resource allocation for machine learned models.

GPU resource usage tracking in machine-learned model execution system using inference wrapper

Disclosed are techniques for combining data using transformer-based machine learning models. In some examples, a first transformer is used to combine a first dataset with a second dataset. The results are then combined with a third dataset, using a second transformer. Each dataset can represent data from a different sensor modality. The transformers compute scores based on queries and apply the scores to values. The first dataset can be used to generate queries for the first transformer, and the values for the first transformer can be derived from the second dataset. Similarly, the third dataset can be used to generate queries for the second transformer, and the values for the second transformer can be derived from the output of the first transformer. The output of the second transformer is therefore a combination of all three datasets and can be used for object detection, for example, determining three-dimensional boundaries of objects.

Object detection using transformer based fusion of multi-modality sensor data

A system for faster object attribute and/or intent classification may include an machine-learned (ML) architecture that processes temporal sensor data (e.g., multiple instances of sensor data received at different times) and includes a cache in an intermediate layer of the ML architecture. The ML architecture may be capable of classifying an object's intent to enter a roadway, idling near a roadway, or active crossing of a roadway. The ML architecture may additionally or alternatively classify indicator states, such as indications to turn, stop, or the like. Other attributes and/or intentions are discussed herein.

Machine-learned architecture for efficient object attribute and/or intention classification

Techniques are described herein for encoding of an input data, to handle both variable bitrate requirements and varying importance of content of different portions of the input data. The encoding vectors may be based on a subset of data from the input data. Potential distortion in the reconstruction on a decoder side may be alleviated by transmitting a difference dataset as a complement to encoding vectors encoded from the input data. The difference dataset may be determined taking into account the importance of content of different portions of the input image to reduce the size, for example by masking out portion of the input data that is considered less important. The difference dataset may be compressed based on an available bandwidth.

Systems and methods for encoding an image

Techniques are disclosed for a covariance model that may generate observation covariances based on observation data of object detections. Techniques may include determining observation data for an object detection of an object represented in sensor data, determining that track data of a track is associated with the object, and inputting the observation data associated with the object detection into a machine-learned model configured to output a covariance (a covariance model). The covariance model may output one or more observation covariance values for the observation data. In some examples, the techniques may include determining updated track data based on the track data, the one or more observation covariance values, and the observation data.

Learned state covariances

Systems and techniques for detecting and locating low impact collisions, such as relatively low energy impact collisions involving an autonomous vehicle, are disclosed herein. Sensor data from audio sensors configured on a vehicle may be used to determine whether a threshold number of such sensors have detected sounds associated with a collision. Scores determined for the sensors based on audio energy and confidence values can be used to determine a location estimate for the detected collision.

Event detection and localization using audio

Techniques for correlating perception system errors with unwanted vehicle behavior. The techniques may include receiving log data associated with a vehicle traversing an environment. Based at least in part on the log data, an error associated with an output received from a perception component of the vehicle may be identified. The output of the perception component may be associated with a detection of an object in the environment. The techniques may also include determining that the error contributed to an unwanted behavior of the vehicle. In some examples, a simulation of a planner component of the vehicle may be ran using ground truth data to determine that the error contributed to the unwanted behavior. Based at least in part on the error contributing to the unwanted behavior, a parameter of the perception component may be updated to minimize the error and the unwanted behavior.

Correlating perception errors with vehicle behavior

An error estimation component may generate contours or a heat map of a likelihood that an object detection or prediction generated by a perception component of an autonomous vehicle is inaccurate. The contours or heat map may be used to lock or unlock actions available to a planning component and the planning component may use the contours or heat for determining a trajectory for controlling the autonomous vehicle. The planning component may additionally or alternatively alter operation of a sensor or determine a trajectory to unlock a target action that is currently locked due to a size/extent of uncertainty indicated by a contour/heat map.

Perception error model for fast simulation and estimation of perception system reliability and/or for control system tuning

Growing community of inventors

Menlo Park, CA, United States of America

Subhasis Das