Detection, tracking and 3d modeling of objects with sparse rgb-d slam and interactive perception

Abstract

An object detection system for detecting and manipulating objects on a workspace includes a three dimensional (3D) sensor configured to acquire and transmit point clouds of a scene, each point cloud including one or more objects in the workspace, manipulator configured to move or grip each of the one or more objects, a memory to store the images and a computer executable program including an object detection-localization program, a segmentation program, a gripping-moving program and a geometry reconstruction program, a processor to perform the computer executable program using the images in connection with the 3D sensor, the memory and the manipulator. The computer executable program is configured to cause the processor to perform steps of generating frames including the point clouds and measurements, generating an initial map from a first frame, moving part of the one or more objects by using the manipulator using the first image, acquiring, after the moving step, a second frame, generating a first dynamic map and a static map from the initial map using the second frame, moving another part of the one or more objects based on the first dynamic map, splitting at least one dynamic map among the first dynamic map and the second dynamic maps when the at least one dynamic map includes a segment being outlier with respect to the one or more objects, reconstructing a 3D geometry of the one or more objects from the split map, and determining when the one or more objects are separated from each other.