Densifying sparse depth maps

Abstract

A system and techniques that use one or more machine learning models to predict a dense depth map (e.g., of depth values for all pixels or at least more pixels than a sparse estimation source (e.g., SLAM)). In some implementations, the machine learning model includes two sub models (e.g., neural networks). The first machine learning model predicts computer vision data such as semantic labels and surface normal directions from an input image. This computer vision data will be used to add to or otherwise improve sparse depth data. Specifically, a second machine learning model takes the semantic labels and surface normal directions from and sparse depth data (e.g., 3D points) from a sparse point estimation source (e.g., SLAM) as inputs and outputs a depth map. The output depth map effectively densities the initial depth data (e.g., from SLAM) by providing depth data for additional pixels of the image.