Zoox, Inc.

Vasiliy Karasev

Sarah Tariq

James William Vaisey Philbin

Kai Zhenyu Wang

Kratarth Goel

Yi Xu

Xin Wang

Oytun Ulutan

Abishek Krishna Akella

Rick Zhang

Juhana Kangaspunta

Tencia Lee

Techniques for detecting attributes and/or gestures associated with pedestrians in an environment are described herein. The techniques may include receiving sensor data associated with a pedestrian in an environment of a vehicle and inputting the sensor data into a machine-learned model that is configured to determine a gesture and/or an attribute of the pedestrian. Based on the input data, an output may be received from the machine-learned model that indicates the gesture and/or the attribute of the pedestrian and the vehicle may be controlled based at least in part on the gesture and/or the attribute of the pedestrian. The techniques may also include training the machine-learned model to detect the attribute and/or the gesture of the pedestrian.

Pedestrian attribute and gesture detection

Techniques for detecting key points associated with objects in an environment are described herein. The techniques may include receiving sensor data representing a portion of an environment in which the vehicle is operating and inputting the sensor data into a machine-learned model. Based on the input sensor data, the machine-learned model may detect one or more key points corresponding to physical features (e.g., hands, feet, eyes, etc.) of a pedestrian who is in the environment. Based on the one or more key points, a bounding box associated with the pedestrian may be generated and the vehicle may be controlled based on at least one of the key points or the bounding box. The techniques may also include training the machine-learned model to detect key points associated with pedestrians.

Key point detection

Machine-learned model training for pedestrian attribute and gesture detection

Techniques are discussed for determining a velocity of an object in an environment from a sequence of images (e.g., two or more). A first image of the sequence is transformed to align the object with an image center. Additional images in the sequence are transformed by the same amount to form a sequence of transformed images. Such sequence is input into a machine learned model trained to output a scaled velocity of the object (a relative object velocity (ROV)) according to the transformed coordinate system. The ROV is then converted to the camera coordinate system by applying an inverse of the transformation. Using a depth associated with the object and the ROV of the object in the camera coordinate frame, an actual velocity of the object in the environment is determined relative to the camera.

Object velocity from images

Techniques for determining a vehicle action and controlling a vehicle to perform the vehicle action for navigating the vehicle in an environment can include determining a vehicle action, such as a lane change action, for a vehicle to perform in an environment. The vehicle can detect, based at least in part on sensor data, an object associated with a target lane associated with the lane change action sensor data. In some instances, the vehicle may determine attribute data associated with the object and input the attribute data to a machine-learned model that can output a yield score. Based on such a yield score, the vehicle may determine whether it is safe to perform the lane change action.

Yield behavior modeling and prediction

A trajectory estimate of a wheeled vehicle can be determined based at least in part on determining a wheel angle associated with the vehicle. In some examples, at least a portion of the image associated with the wheeled vehicle may be input into a machine-learned model that is trained to classify and/or regress wheel directions of wheeled vehicles. The machine-learned model may output a predicted wheel direction. The wheel direction and/or additional or historical sensor data may be used to estimate a trajectory of the wheeled vehicle. The predicted trajectory of the object can then be used to generate and refine an autonomous vehicle's trajectory as the autonomous vehicle proceeds through the environment.

Object trajectory from wheel direction

Techniques for determining and/or predicting a trajectory of an object by using the appearance of the object, as captured in an image, are discussed herein. Image data, sensor data, and/or a predicted trajectory of the object (e.g., a pedestrian, animal, and the like) may be used to train a machine learning model that can subsequently be provided to, and used by, an autonomous vehicle for operation and navigation. In some implementations, predicted trajectories may be compared to actual trajectories and such comparisons are used as training data for machine learning.

Motion prediction based on appearance

An object position and/or orientation can be determined based on image data and object contact points. Image data can be captured representing an object, such as a vehicle. Vehicle contact points can be identified in the image data representing wheel contacts with the ground. For an individual vehicle contact point (e.g., a left-front wheel of the second vehicle), a ray can be determined that emanates from the image sensor and passes through the vehicle contact point. To determine a location and velocity of the vehicle, the ray can be unprojected onto a three-dimensional surface mesh, and an intersection point between the ray and the three-dimensional surface mesh can be used as an initial estimate for the projected location of the vehicle contact point in the world. The estimated location can be adjusted based on various cost functions to optimize an accuracy of the locations of the estimated vehicle contact points.

Growing community of inventors

San Francisco, CA, United States of America

Vasiliy Karasev