Zoox, Inc.

Derek Xiang Ma

Subhasis Das

Oytun Ulutan

Sarah Tariq

Adam Pollack

Amir Ghaderi

Yi-Ting Lin

Jesse Sol Levinson

Zeng Wang

Kratarth Goel

Scott M Purdy

Zejia Zheng

Bryce A Evans

Ekaterina Hristova Taralova

Justin Nordgreen

Jonathan Tyler Dowdall

Kenneth Michael Siebert

Soroush Dean Khadem

William Harland Montgomery, Iv

Hang Ren

Shuangyu Li

Techniques for detecting and labeling construction zones in an environment are disclosed. Two-dimensional images may be evaluated to identify pixels that may be associated with a construction zone and labeled accordingly. Labels for corresponding pixels in a separate image or non-construction zone labels for the same pixels may be compared to the construction zone pixels and an output image pixel label may be determined based on one or more criteria. An output image can be provided for vehicle control and for other operations, such as top-down segmentation and trajectory determination. Output images and related data may also be used to train a model to perform construction zone detection and labeling.

Systems and methods for construction zone detection

Techniques are discussed herein for controlling autonomous vehicles within a driving environment, including generating and using bounding contours associated with objects detected in the environment. Image data may be captured and analyzed to identify and/or classify objects within the environment. Image-based and/or lidar-based techniques may be used to determine depth data associated with the objects, and a bounding contour may be determined based on the object boundaries and associated depth data. An autonomous vehicle may use the bounding contours of objects within the environment to classify the objects, predict the positions, poses, and trajectories of the objects, and determine trajectories and perform other vehicle control actions while safely navigating the environment.

Object bounding contours based on image data

Techniques for detecting and labeling traffic lights and/or traffic signs and the traffic lanes with which such lights and signs are associated in an environment are disclosed. Images may be evaluated to identify pixels that may be associated with a light, sign, or lane. Associations between lights and/or signs and lanes in the environment may be determined along with probabilities for the individual pixels that the pixels may be associated with one of the light/sign and lane associations. Those pixels having a sufficient probability of be associated with a light/sign and lane association may be assigned a corresponding label. An output image with such labels can be provided for vehicle control and for other operations, such as top-down segmentation and trajectory determination.

Systems and methods for traffic light detection

Techniques for adjusting vehicle models based on environmental conditions are discussed herein. The techniques may include receiving image data representing a portion of an environment in which a vehicle is operating and inputting the image data into a machine learned model. Additionally, data representing an environmental condition associated with the environment may be received from a sensor of the vehicle to detect changes in the environmental conditions such that one or more actions associated with the machine learned model or an output of the machine learned model may be performed. Some of the techniques may also include running multiple machine learned models or multiple configurations of a machine learned model in parallel and selecting different outputs of the machine learned model(s) to adjust for changes in the environmental conditions. For instance, individual outputs may be selected based on environmental conditions, confidence scores, thresholds, etc.

Adjusting vehicle models based on environmental conditions

Techniques are described herein for determining a yaw value for a traffic light head. The technique comprises obtaining sensor data via the sensor associated with the vehicle, the sensor data including a traffic light head. Based at least in part on the sensor data, a yaw value is determined for the traffic light head indicative of a degree that the traffic light head is yawed with respect to a line of sight of the sensor or the vehicle. Based at least in part on the yaw value, an extent that the traffic light head is associated with navigation of the vehicle is determined. The vehicle is then controlled based at least in part on the extent that the traffic light head is associated with the navigation of the vehicle.

Yawed traffic light determination

Techniques are disclosed for tracking objects in sensor data, such as multiple images or multiple LIDAR clouds. The techniques may include comparing segmentations of sensor data such as by, for example, determining a similarity of a first segmentation of first sensor data and a second segmentation of second sensor data. Comparing the similarity may comprise determining a first embedding associated with the first segmentation and a second embedding associated with the second segmentation and determining a distance between the first embedding and the second embedding. The techniques may improve the accuracy and/or safety of systems integrating the techniques discussed herein.

Tracking objects using sensor data segmentations and/or representations

Parallel vehicle model processing

Techniques for segmenting and classifying a representation of aggregated sensor data from a scene are discussed herein. Sensor data may be collected during multiple traversals of a same scene, and the sensor data may be filtered to remove portions of the sensor data not relevant to road network maps. In some examples, the filtered data may be aggregated and represented in voxels of a three-dimensional voxel space, from which an image representing a top-down view of the scene may be generated, though other views are also contemplated. Operations may include segmenting and/or classifying the image e.g., by a trained machine-learned model, to associate class labels indicative of map elements (e.g., driving lane, stop line, turn lane, and the like) with segments identified in the image. Additionally, techniques may create or update road network maps based on segmented and semantically labeled image(s) of various portions of an environment.

Semantic segmentation of aggregated sensor data

Techniques for detecting, locating, and/or classifying objects based on multiple sensor data inputs received from different sensor modalities. The techniques may include receiving sensor data generated by different sensor modalities of a vehicle, the sensor data including at least first sensor data generated by a first sensor modality and second sensor data generated by a second sensor modality. In some examples, the sensor data may be input into a machine-learning pipeline. The machine-learning pipeline may be configured to determine locations of objects in an environment surrounding the vehicle based at least in part on a correlation, by the multi-attention component, of the first sensor data and the second sensor data. The techniques may also include receiving, from the machine-learning pipeline, an output indicating a location of an object in the environment.

Multi-attention machine learning for object detection and classification

A system for faster object attribute and/or intent classification may include an machine-learned (ML) architecture that processes temporal sensor data (e.g., multiple instances of sensor data received at different times) and includes a cache in an intermediate layer of the ML architecture. The ML architecture may be capable of classifying an object's intent to enter a roadway, idling near a roadway, or active crossing of a roadway. The ML architecture may additionally or alternatively classify indicator states, such as indications to turn, stop, or the like. Other attributes and/or intentions are discussed herein.

Machine-learned architecture for efficient object attribute and/or intention classification

There is provided methods, systems, and computer-readable media for determining intention of bicycles and other person-wide vehicles. A method comprises: receiving, from one or more sensors of an autonomous vehicle, sensor data relating to an external environment of the autonomous vehicle; detecting, based at least in part on the sensor data, a person-wide vehicle in the external environment proximate to the autonomous vehicle; determining, based at least in part on the sensor data and the person-wide vehicle, image data including the person-wide vehicle; inputting the image data to a machine-learned model; receiving, from the machine-learned model based on the image data, a future intention of the person-wide vehicle; and controlling the autonomous vehicle based at least in part on the future intention of the person-wide vehicle.

Determining intention of bicycles and other person-wide vehicles

Techniques are described for detecting whether a lane of a roadway is open or closed. Detecting a lane as being closed may include detecting an object in or near the lane, which may comprise determining a size, location, and/or classification associated with the object, and dilating the size associated with the object. The lane may be indicated as being closed if a distance between a dilated object detection and another object detection, dilated object detection, or lane extent is less than a threshold distance. The techniques may additionally or alternatively comprise determining an alternative lane shape based at least in part on one or more object detections and/or determining that one or more lanes are closed and/or uploading a lane closure and/or alternative lane shape to a central database for retrieval by/dissemination to other computing devices.

Growing community of inventors

Redwood City, CA, United States of America

Derek Xiang Ma