Pony Ai Inc.

Pony Al Inc.

Novumind Limited

Pingfan Meng

Zhenhao Pan

Yubo Zhang

Kai Chen

Stephen Lee

Michael C Wu

Brian James Rush

Wei-Yang Chiu

Yilei Li

Cheng-Ying Ko

Jeffrey Weifung Lien

Miao Li

A computing device performs initial processing of sensor data. The computing device performs obtaining sensor data, writing the sensor data to first addresses of a dynamically allocated buffer associated with the computing device, encoding the sensor data, writing the encoded sensor data to second addresses of the dynamically allocated buffer, in response to completing the writing of the encoded sensor data, indicating that the writing of the encoded sensor data has been completed, receiving, from a computing resource, a polling request to read the encoded sensor data, transmitting, to the computing resource, a status that the writing of the encoded sensor data to the second addresses has been completed, reading, to a memory of the computing resource, the encoded sensor data, receiving, from the computing resource, a second status that the encoded sensor data has been read, and removing, from the dynamically allocated buffer, the encoded sensor data.

Efficient retrieval of sensor data while ensuring atomicity

Provided herein is a system and method for determining whether a sensor is calibrated and recalibrating of an uncalibrated sensor. The system comprises a sensor system comprising a sensor and an analysis engine configured to determine whether the sensor is uncalibrated. The system further comprises an error handling system configured to determine whether to perform a recalibration in response to the sensor system determining that the sensor is uncalibrated. The error handling system further comprises a recalibration engine configured to perform a recalibration.

System and method for recalibration of an uncalibrated sensor

A computing device performs initial processing of sensor data. The computing device includes one or more processors and instructions or logic that, when executed by the one or more processors, cause the computing device to perform obtaining sensor data, encoding the sensor data, writing the encoded sensor data to a dynamically allocated buffer, and logging a status of the written encoded sensor data at a static location of the dynamically allocated buffer. The status includes any one or more of memory addresses at which frames of the sensor data begin in the dynamically allocated buffer, valid bit fields corresponding to the frames, and sizes of each of data segments within the frames. The instructions further cause the computing device to perform, in response to receiving a polling request from a computing resource, transmitting the logged status to the computing resource.

Efficient retrieval of sensor data

Described herein are systems, methods, and non-transitory computer-readable media for self-detection of a fault condition by a vehicle component, generation of a device health code that includes multiple tiers of information relating to the fault condition experienced by the vehicle component, and broadcasting of the device health code to one or more other vehicle components via one or more vehicle communication networks. A recommended vehicle response measure indicated by a reaction code in the device health code can then be taken or alternate vehicle response may be selected and initiated based on an evaluation of current vehicle operational data.

Device-level fault detection

Described herein are systems, methods, and non-transitory computer-readable media for isolating commercial components from a harsh vehicle operating environment to increase the longevity of such components and to decrease their failure rate. Also described herein are systems, methods, and non-transitory computer-readable media for monitoring the operational health status of vehicle components for failure, and upon detecting failure of a component, initiating a processing task reassignment and fault recovery process. In this manner, processing load handled by the component prior to failure can be offloaded to one or more other vehicle components while a fault recovery process is initiated for the component. When the failed component is operational again, the vehicle may revert back to the task assignment in place prior to the component failure, may continue with the current task assignment, or may transition to another different task reassignment.

Autonomous driving vehicle health monitoring

Provided herein is a system and method for determining whether a sensor is calibrated and error handling of an uncalibrated sensor. The system comprises a sensor system comprising a sensor and an analysis engine configured to determine whether the sensor is uncalibrated. The system further comprises an error handling system configured to perform an error handling in response to the sensor system determining that the sensor is uncalibrated. The method comprises determining, by a sensor system, whether the sensor is uncalibrated, and performing, by an error handling system, an error handling in response to the sensor system determining that the sensor is uncalibrated.

System and method for error handling of an uncalibrated sensor

Described herein are systems, methods, and non-transitory computer-readable media for self-detection of fault conditions experienced by vehicle components, generation of device health codes indicative of the fault conditions, and broadcasting of the device health codes over mixed vehicle communication networks. The device health codes can be parsed to identify fault information, and the fault information can be assessed along with current vehicle operational data to determine a recommended vehicle response measure to one or more fault conditions experienced by one or more vehicle components.

Device health code broadcasting on mixed vehicle communication networks

A system comprises a sensor system comprising a sensor and an analysis engine configured to determine whether the sensor is uncalibrated. The system further comprises an error handling system configured to determine whether to perform a recalibration in response to the sensor system determining that the sensor is uncalibrated. The error handling system further comprises a recalibration engine configured to perform a recalibration.

Described herein are systems, methods, and non-transitory computer readable media for memory address encoding of multi-dimensional data in a manner that optimizes the storage and access of such data in linear data storage. The multi-dimensional data may be spatial-temporal data that includes two or more spatial dimensions and a time dimension. An improved memory architecture is provided that includes an address encoder that takes a multi-dimensional coordinate as input and produces a linear physical memory address. The address encoder encodes the multi-dimensional data such that two multi-dimensional coordinates close to one another in multi-dimensional space are likely to be stored in close proximity to one another in linear data storage. In this manner, the number of main memory accesses, and thus, overall memory access latency is reduced, particularly in connection with real-world applications in which the respective probabilities of moving along any given dimension are very close.

Memory architecture for efficient spatial-temporal data storage and access

Described herein are systems, methods, and non-transitory computer readable media for constructing and utilizing vehicle field-of-view (FOV) information. The FOV information can be utilized in connection with vehicle localization such as localization of an autonomous vehicle (AV), sensor data fusion, or the like. A customized computing machine can be provided that is configured to construct and utilize the FOV information. The customized computing machine can utilize the FOV information, and more specifically, FOV semantics data included therein to manage various data and execution patterns relating to processing performed in connection with operation of an AV such as, for example, data prefetch operations, reordering of sensor data input streams, and allocation of data processing among multiple processing cores.

Systems and methods for constructing and utilizing field-of-view (FOV) information

Described herein are systems, methods, and computer readable media for performing data conversion on sensor data to obtain modified sensor data that is formatted/structured appropriately for downstream processes that rely on the sensor data as input. The sensor data can include point cloud data captured by a LiDAR, for example. A grid structure and corresponding grid characteristics can be determined and the sensor data can be converted to grid-based sensor data by associating the grid structure and its characteristics with the sensor data. Generating the grid-based sensor data can include reformatting the point cloud data to superimpose the grid structure and its grid characteristics onto the point cloud data. Various downstream processing that cannot feasibly be performed on the raw sensor data can then be performed efficiently on the modified grid-based sensor data by virtue of the grid structure imbuing the sensor data with spatial proximity information.

Point cloud data reformatting

Described herein is a memory architecture that is configured to dynamically determine an address encoding to use to encode multi-dimensional data such as multi-coordinate data in a manner that provides a coordinate bias corresponding to a current memory access pattern. The address encoding may be dynamically generated in response to receiving a memory access request or may be selected from a set of preconfigured address encodings. The dynamically generated or selected address encoding may apply an interleaving technique to bit representations of coordinate values to obtain an encoded memory address. The interleaving technique may interleave a greater number of bits from the bit representation corresponding to the coordinate direction in which a coordinate bias is desired than from bit representations corresponding to other coordinate directions.

Dynamic memory address encoding

Described herein are systems, methods, and non-transitory computer readable media for triggering a sensor operation of a second sensor (e.g., a camera) based on a predicted time of alignment with a first sensor (e.g., a LiDAR), where operation of the second sensor is simulated to determine the predicted time of alignment. In this manner, the sensor data captured by the two sensors is ensured to be substantially synchronized with respect to the physical environment being sensed. This sensor data synchronization based on predicted alignment of the sensors solves the technical problem of lack of sensor coordination and sensor data synchronization that would otherwise result from the latency associated with communication between sensors and a centralized controller and/or between sensors themselves.

Sensor triggering based on sensor simulation

Described herein are systems, methods, and non-transitory computer readable media for performing an alignment between a first vehicle sensor and a second vehicle sensor. Two-dimensional (2D) data indicative of a scene within an environment being traversed by a vehicle is captured by the first vehicle sensor such as a camera or a collection of multiple cameras within a sensor assembly. A three-dimensional (3D) representation of the scene is constructed using the 2D data. 3D point cloud data also indicative of the scene is captured by the second vehicle sensor, which may be a LiDAR. A 3D point cloud representation of the scene is constructed based on the 3D point cloud data. A rigid transformation is determined between the 3D representation of the scene and the 3D point cloud representation of the scene and the alignment between the sensors is performed based at least in part on the determined rigid transformation.

Growing community of inventors

Dublin, CA, United States of America

Pingfan Meng