Streem, Inc.

Nikilesh Urella

Flora Ponjou Tasse

Ghislain Tasse

Embodiments include systems, processes, and/or techniques for creating a semantically segmented 3D mesh of a physical environment, where the semantically segmented 3D mesh may be at least partially created by a first user, and where a second user, for example a service professional, may view and modify one of the segments of the semantically segmented 3D mesh for subsequent viewing by the first user. Other embodiments may be described and/or claimed.

Modifying a portion of a semantically segmented 3D mesh

Embodiments of devices and techniques of obtaining a three dimensional (3D) representation of an area are disclosed. In one embodiment, a two dimensional (2D) frame is obtained of an array of pixels of the area. Also, a depth frame of the area is obtained. The depth frame includes an array of depth estimation values. Each of the depth estimation values in the array of depth estimation values corresponds to one or more corresponding pixels in the array of pixels. Furthermore, an array of confidence scores is generated. Each confidence score in the array of confidence scores corresponds to one or more corresponding depth estimation values in the array of depth estimation values. Each of the confidence scores in the array of confidence scores indicates a confidence level that the one or more corresponding depth estimation values in the array of depth estimation values is accurate.

Confidence score for depth map or 3D mesh

Frames for texturing a 3D mesh may be selected to minimize the number of frames required to completely texture the mesh, thus reducing the overhead of texturing. Keyframes are selected from a video stream on the basis of amount of overlap from previously selected keyframes, with the amount of overlap held below a predetermined threshold. The 3D mesh may also be refined and corrected to ensure a higher quality mesh application, including color correction of the selected keyframes. Other embodiments are described.

Efficient texture mapping of a 3-D mesh

A mesh model of a 3D space is provided with improved accuracy by refining the locations of edges of objects in the space. The mesh model includes vertices which define surfaces of triangles. Triangles are identified which have two vertices in one plane and another, outlier vertex in another, adjacent plane. A line is fitted to the outlier vertices to define an edge of an object, and the outlier vertices are moved to the line, referred to as a mesh-based line. Texture data from images of the space can be used to further refine the edge. In one approach, gradients in grayscale pixels which correspond the vertices of the mesh-based line are used to define a grayscale-based line. The two line definitions can be combined or otherwise used to provide a final definition of the edge. The object can be measured based on the length and position of the edge.

Edge enhancement with texture optimization

Embodiments herein may relate to a technique to be performed by surface partition logic. The technique may include identifying a first mesh portion that is related to a first plane of a three-dimensional (3D) space, and a second mesh portion that is related to a second plane of the 3D space. The technique may include identifying, based on a linear representation of a border between the first mesh portion and the second mesh portion, an element of the first mesh portion that at least partially overlaps the second mesh portion. The technique may further include altering the element of the first mesh portion to reduce the amount that the element overlaps the second mesh portion. Other embodiments may be described and/or claimed.

Surface partition with plane and line constraints

A sequence of frames including color and depth data is processed to identify key frames while minimizing redundancy. A sparse 3D point cloud is obtained for each frame and represented by a set of voxels. Each voxel has associated data indicating, e.g., a depth and a camera viewing angle. When a new frame is processed, a new sparse 3D point cloud is obtained. For points which are not encompassed by the existing voxels, new voxels are created. For points which are encompassed by the existing voxels, a comparison determines whether the depth data of the new frame is more accurate than the existing depth data. A frame is selected as a key frame based on factors such as a number of new voxels which are created, a number of existing voxels for which the depth data is updated, and accuracy scores.

Growing community of inventors

Portland, OR, United States of America

Nikilesh Urella