Dynamic actuation map using neural network fed by visual odometry

Abstract

A method of controlling actuators of a vehicle includes receiving pixel data corresponding to an area outside of the vehicle, receiving desired dynamic set-points for the vehicle from a path planning application of the vehicle, receiving sensor data from the vehicle, generating one or more real actuator commands for the vehicle in response to the sensor data from the vehicle, the desired dynamic set-points for the vehicle received from the path planning application of the vehicle, and one or more inferences made by a first trained neural network model using the sensor data from the vehicle and the desired dynamic set-points for the vehicle as input, and communicating the one or more real actuator commands for the vehicle to actuators of the vehicle. The one or more real actuator commands for the vehicle are generally presented at an output of the first trained neural network model. The first trained neural network model was generally trained by performing computer vision operations on the pixel data corresponding to the area outside the vehicle arranged as video frames to detect features in a first set of the video frames, determining a first set of measured dynamic set-points for the vehicle by applying visual odometry operations on the features detected in the first set of the video frames, and modifying a plurality of weights and bias values of an untrained neural network model based on the first set of measured dynamic set-points, the sensor data, and a training dataset comprising dynamic set-points and corresponding actuator values that are representative of one or more desired inferences.