Generating 3d representations of physical environments using key frames

Abstract

Various implementations disclosed herein include devices, systems, and methods that use frames of sensor data to create key frames that are used to provide a 3D representation of a physical environment (e.g., a 3D mesh, planar representations, object bounding boxes, a combo of these, etc.). Some implementations use key frames (e.g., bundles of 2D image data deemed to be representative or useful for reconstruction/3D representation generation). Such key frames may be created by evaluating multiple frames of sensor data and determining whether to convert a given frame into a key frame or fuse it, e.g., fuse it with a nearest existing key frame. Such fusing may reduce loss of information which would occur if frames not selected as key frames are simply discarded.