Zoox, Inc.

Sarah Tariq

Kratarth Goel

James William Vaisey Philbin

Vasiliy Karasev

Yi Xu

Kai Zhenyu Wang

Derek Xiang Ma

Praveen Srinivasan

Qijun Tan

Bryce A Evans

Chuang Wang

Marc Wimmershoff

Feng Tian

Cooper Stokes Sloan

Xin Wang

Oytun Ulutan

Jesse Sol Levinson

Subasingha Shaminda Subasingha

Adam Pollack

Zejia Zheng

Ekaterina Hristova Taralova

Justin Nordgreen

Pengfei Duan

Thomas Oscar Dudzik

Venkata Subrahmanyam Chandra Sekhar Chebiyyam

Ravi Verma Gogna

Michael Carsten Bosse

Jefferson Bradfield Packer

Nam Gook Cho

Noureldin Ehab Hendy

Jonathan Tyler Dowdall

Tencia Lee

Amanda Brown Prescott

Dilip Bethanabhotla

Techniques for detecting and responding to an emergency vehicle are discussed. A vehicle computing system may determine that an emergency vehicle based on sensor data, such as audio and visual data. In some examples, the vehicle computing system may determine aggregate actions of objects (e.g., other vehicles yielding) proximate the vehicle based on the sensor data. In such examples, a determination that the emergency vehicle is operating may be based on the actions of the objects. The vehicle computing system may, in turn, identify a location to move out of a path of the emergency vehicle (e.g., yield) and may control the vehicle to the location. The vehicle computing system may determine that the emergency vehicle is no longer relevant to the vehicle and may control the vehicle along a route to a destination. Determining to yield and/or returning to a mission may be confirmed by a remote operator.

Emergency vehicle detection and response

Techniques for adjusting vehicle models based on environmental conditions are discussed herein. The techniques may include receiving image data representing a portion of an environment in which a vehicle is operating and inputting the image data into a machine learned model. Additionally, data representing an environmental condition associated with the environment may be received from a sensor of the vehicle to detect changes in the environmental conditions such that one or more actions associated with the machine learned model or an output of the machine learned model may be performed. Some of the techniques may also include running multiple machine learned models or multiple configurations of a machine learned model in parallel and selecting different outputs of the machine learned model(s) to adjust for changes in the environmental conditions. For instance, individual outputs may be selected based on environmental conditions, confidence scores, thresholds, etc.

Adjusting vehicle models based on environmental conditions

Techniques are disclosed for tracking objects in sensor data, such as multiple images or multiple LIDAR clouds. The techniques may include comparing segmentations of sensor data such as by, for example, determining a similarity of a first segmentation of first sensor data and a second segmentation of second sensor data. Comparing the similarity may comprise determining a first embedding associated with the first segmentation and a second embedding associated with the second segmentation and determining a distance between the first embedding and the second embedding. The techniques may improve the accuracy and/or safety of systems integrating the techniques discussed herein.

Tracking objects using sensor data segmentations and/or representations

Parallel vehicle model processing

Techniques for detecting attributes and/or gestures associated with pedestrians in an environment are described herein. The techniques may include receiving sensor data associated with a pedestrian in an environment of a vehicle and inputting the sensor data into a machine-learned model that is configured to determine a gesture and/or an attribute of the pedestrian. Based on the input data, an output may be received from the machine-learned model that indicates the gesture and/or the attribute of the pedestrian and the vehicle may be controlled based at least in part on the gesture and/or the attribute of the pedestrian. The techniques may also include training the machine-learned model to detect the attribute and/or the gesture of the pedestrian.

Pedestrian attribute and gesture detection

Techniques for verifying a reliability of map data are discussed herein. In some examples, map data can be used by a vehicle, such as an autonomous vehicle, to traverse an environment. Sensor data (e.g., image data, lidar data, etc.) can be received from a sensor associated with a vehicle and may be used to generate an estimated map and confidence values associated with the estimated map. When the sensor data is image data, images data from multiple perspectives or different time instances may be combined to generate the estimated map. The estimated map may be compared to a stored map or to a proposed vehicle trajectory or corridor to determine a reliability of the stored map data.

Verifying reliability of data used for autonomous driving

Techniques are disclosed for implementing a convolutional neural network that determines an offset field for deforming a kernel to be used in a convolution. The offset field is temporally-based, at least in part, on data generated at an earlier time. Furthermore, techniques are disclosed for using sensor data to train a neural network to learn shapes or configurations of such deformed kernels. The temporal-based deformable convolutions may be used for object identification, object matching, object classification, segmentation, and/or object tracking, in various examples.

Temporal-based deformable kernels

Techniques relating to monitoring map consistency are described. In an example, a monitoring component associated with a vehicle can receive sensor data associated with an environment in which the vehicle is positioned. The monitoring component can generate, based at least in part on the sensor data, an estimated map of the environment, wherein the estimated map is encoded with policy information for driving within the environment. The monitoring component can then compare first information associated with a stored map of the environment with second information associated with the estimated map to determine whether the estimated map and the stored map are consistent. Component(s) associated with the vehicle can then control the object based at least in part on results of the comparing.

Map consistency checker

Techniques for detecting key points associated with objects in an environment are described herein. The techniques may include receiving sensor data representing a portion of an environment in which the vehicle is operating and inputting the sensor data into a machine-learned model. Based on the input sensor data, the machine-learned model may detect one or more key points corresponding to physical features (e.g., hands, feet, eyes, etc.) of a pedestrian who is in the environment. Based on the one or more key points, a bounding box associated with the pedestrian may be generated and the vehicle may be controlled based on at least one of the key points or the bounding box. The techniques may also include training the machine-learned model to detect key points associated with pedestrians.

Key point detection

Machine-learned model training for pedestrian attribute and gesture detection

A machine-learning (ML) architecture for determining three or more outputs, such as a two and/or three-dimensional region of interest, semantic segmentation, direction logits, depth data, and/or instance segmentation associated with an object in an image. The ML architecture may output these outputs at a rate of 30 or more frames per second on consumer grade hardware.

Multi-task learning for real-time semantic and/or depth aware instance segmentation and/or three-dimensional object bounding

Techniques for utilizing microphone or audio data to detect and responding to low velocity impacts to a system such as an autonomous vehicle. In some cases, the system may be equipped with a plurality of microphones that may be used to detect impacts that fail to register on the data captured by the vehicle's inertial measurement units and may go undetected by the vehicle's perception system and sensors. In one specific example, the perception system of the autonomous vehicle may identify a period of time in which a potential low velocity impact may occur. The autonomous vehicle may then utilize the microphone or audio data associated with the period of time to determine if an impact occurred.

System to detect impacts

Techniques for maintaining and synchronizing data is a processing pipeline data between multiple processing units to improve a system latency are described herein. For example, the techniques may include determining, in response to an invocation of vision processing on first vision data stored in a first memory range in a first memory associated with a central processing unit (CPU), that second vision data stored in a second memory range in a second memory associated with a graphic processing unit (GPU) is a modified copy of the first vision data. The second vision data may be obtained using a non-blocking operation from the second memory range. The first vision data stored in the first memory range may be replaced with the second vision data obtained from the second memory range. The vision processing may then be performed using the second vision data stored in the first memory.

Vision architecture

A vehicle computing system may implement techniques to determine attributes (or intent) of an object detected by a vehicle operating in the environment. The techniques may include determining a set of features with respect to a detected object by a first model and determining, by a second model and based on the set of features, one or more attributes of the object. The first model and the second model may be configured to process at least one image frame to determine the one or more attributes of the object. A model may receive sensor data as an input, and output features and/or an attribute for the detected object. Based on the attribute(s) of the object, a vehicle computing system may control operation of the vehicle.

Object or person attribute characterization

Techniques are discussed for determining a velocity of an object in an environment from a sequence of images (e.g., two or more). A first image of the sequence is transformed to align the object with an image center. Additional images in the sequence are transformed by the same amount to form a sequence of transformed images. Such sequence is input into a machine learned model trained to output a scaled velocity of the object (a relative object velocity (ROV)) according to the transformed coordinate system. The ROV is then converted to the camera coordinate system by applying an inverse of the transformation. Using a depth associated with the object and the ROV of the object in the camera coordinate frame, an actual velocity of the object in the environment is determined relative to the camera.

Growing community of inventors

Palo Alto, CA, United States of America

Sarah Tariq