Zoox, Inc.

James Graham Dolan

Anas Lasram

Douglas Raymond Brooks

Tod Cameron Semple

Chuang Wang

Ryan Martin Cahoon

Leonardo Poubel Orenstein

Matthew Lotocki

Takahiro Baba

Ximing Li

A simulation system may generate radar data for synthetic simulations of autonomous vehicles by determining and applying radar masks for object radar data based on the attributes of simulated objects. A simulation system may determine a simulated object within a region of a simulated environment, and may apply a radar mask to determine the radar data within the region based on attributes of the simulated object. Such attributes may include the material properties, object type, and/or distance of the simulated object from the simulation sensors. Radar masks may be determining using masking percentages based on combinations of object attributes, and masks may be applied individually or by combining multiple masks for multiple occluding objects, in order to more accurately simulate the behavior of real-world radar signals in autonomous vehicle simulations.

Radar data masks in autonomous vehicle simulations

A controller may use a timing model to control and coordinate the execution of components within a data processing pipeline. The timing model may be generated based on log data collected during execution of the components, to model component latencies and pipeline behavior in a production environment. The controller may use the timing model to control the timing of the inputs to, the outputs from, and the execution of various interacting components in the data processing pipeline. The timing model may include component-specific and/or channel-specific timing data, and in various implementations may be based on inputs provided to the components, the timing and contents of outputs from the components, and/or the number of times particular code is executed within the components.

Modeling component latencies in data processing pipelines

Driving simulations may be generated based on driving log data and used to validate autonomous vehicle safety systems. A driving simulation system may receive driving log data from autonomous vehicles and determine events which caused a vehicle to diverge from a planned trajectory. Driving simulations may be generated based on the driving log data, and executed using a simulated vehicle that follows the initial planned trajectory of the autonomous vehicle. The driving simulations may be analyzed to detect potential collisions and near misses, and to determine success conditions for the behaviors of vehicle safety systems. Driving simulations and associated success conditions may be aggregated and used in validation suites for vehicle controllers and/or vehicle safety systems, thereby providing more comprehensive and robust autonomous vehicle validation based on real-world driving scenarios.

Autonomous vehicle safety system validation

Simulating output of a perception system may comprise receiving scenario data indicating a position associated with a simulated sensor and a position and/or identifier of an object, and instantiating a three-dimensional representation of an environment and the object (i.e., a simulated environment). The system may generate depth data indicating distances and/or positions of surfaces in the simulated environment relative to the simulated sensor position and determine a three-dimensional region of interest based at least in part on the depth data associated with at least a portion of the object. In some examples, the three-dimensional region of interest may be smaller than a size of the object, due to an occlusion by topology of the simulated environment and/or another object in the simulated environment.

Simulating object occlusions

Procedural world generation using tertiary data is described. In an example, a computing device can receive road network data associated with the real environment and a road mesh associated with a real environment. The computing device can associate the road network data with the road mesh to generate a simulated environment. Additionally, the computing device can associate supplemental data with the road network data and the road mesh to enhance the simulated environment (e.g., supplementing information otherwise unavailable to the sensor data due to an occlusion). The computing device can output the simulated environment for at least one of testing, validating, or training an algorithm used by an autonomous robotic computing device for at least one of navigating, planning, or decision making.

Procedural world generation using tertiary data

Techniques associated with improving performance and realism of simulation instances associated with simulation testing of autonomous vehicles. In some cases, a simulation system may be configured to run a pre-simulation test to identify and store occlusion data to improve the performance of subsequent simulations associated with a shared scene or route.

System processing scenario objects during simulation

A sensor simulation system may generate sensor data for use in simulations by rendering two-dimensional views of a three-dimensional simulated environment. In various examples, the sensor simulation system uses sensor dependency data to determine specific views to be re-rendered at different times during the simulation. The sensor simulation system also may generate unified views with multi-sensor data at each region (e.g., pixel) of the two-dimensional view for consumption by different sensor types. A hybrid technique may be used in some implementations in which rasterization is used to generate a view, after which ray tracing is used to align the view with a particular sensor. Spatial and temporal upsampling techniques also may be used, including depth-aware and velocity-aware analyses for simulated objects, to improve view resolution and reduce the frequency of re-rendering views.

Sensor simulation with unified multi-sensor views

An object model for a real-world object is constructed using a parameter-based object generator, where the parameters for the object model are determined using sensor data such as photographic images, radar images, or LIDAR images obtained from the real-world object. Conversion of the sensor data to appropriate object parameters is accomplished using a machine learning system. The machine learning system is calibrated by generating a plurality of object parameters, generating a corresponding plurality of learning objects from the plurality of object parameters, generating a plurality of simulated sensor return data for the learning objects, and then providing the simulated sensor return data and object parameters to the machine learning system.

System and method for modeling physical objects in a simulation

Simulation view generation based on simulated sensor operations

Spatial and temporal upsampling techniques for simulated sensor data

A system for converting global shutter image data into simulated rolling shutter image data may include a global-to-rolling shutter image converter. The converter may be configured to receive global shutter image data associated with a plurality of global shutter images. The global shutter image data may include dynamic global shutter image data representative of a moving object in the plurality of global shutter images. The converter may also be configured to determine an optical flow field based at least in part on two or more of the plurality of global shutter images and generate the simulated rolling shutter image based at least in part on the optical flow field. The converter may also be configured to generate high dynamic range (HDR) images based at least in part on the optical flow field and the global shutter image data.

Simulated rolling shutter image data

Mapping simulated sensors to simulated environment views

A driving simulation system may perform simulations using one or more of a virtual driving simulator or a log-based driving simulator. A log-based simulator may replay data from logs to test whether a control algorithm is successfully capable of navigating a scenario. However, when such log-based simulations are invalid (e.g., based on the results of the simulations or evaluations of conditions of the simulation), a virtual simulation may be generated for the vehicle control system during which new log data may be captured. Such a virtual simulation may comprise sensor simulations and more sophisticated object control for ensuring convergence to a scenario to be tested. The new log data may be used for additional log-based simulations, thereby improving the durability and flexibility for testing and/or validation, while reducing the computational overhead of driving simulation scenarios.

Autonomous driving simulations based on virtual simulation log data

A system for generating simulated LiDAR data may include a depth offset generator and an intensity value generator. The depth offset generator may be configured to receive environment data including depth information, e.g., a depth map, the environment. The depth offset generator may determine an optical flow field from the depth information and estimate depths for positions to simulate LiDAR sensor data. field. The depth offset generator can also generate timestamp information based on attributes of the simulated LiDAR sensor, and determine the estimated depths using the timestamp information. The intensity value generator may be configured to determine an intensity for pixels based on physical attributes associated with those pixels. The simulated LiDAR data may be used in simulations run on autonomous vehicle control systems.

Growing community of inventors

Foster City, CA, United States of America

James Graham Dolan