Streem, Inc.

Pavan Kumar Kamaraju

Flora Ponjou Tasse

Ghislain Fouodji Tasse

Sean M Adkinson

Ryan R Fink

Nikilesh Urella

Renganathan Veerasubramanian

Nicholas Degroot

Ghislain Tasse

Embodiments include systems, processes, and/or techniques for creating a semantically segmented 3D mesh of a physical environment, where the semantically segmented 3D mesh may be at least partially created by a first user, and where a second user, for example a service professional, may view and modify one of the segments of the semantically segmented 3D mesh for subsequent viewing by the first user. Other embodiments may be described and/or claimed.

Modifying a portion of a semantically segmented 3D mesh

Embodiments include systems and methods for generating a 3D mesh on a server from one or more images and other data of a physical space that is captured by a device. The device may capture images with RGB data that includes depth information for pixels of the one or more images. The server may then create or update a 3D mesh based upon the images and depth information. The server may also use received images to provide textures to the various surfaces of the 3D mesh. The server may then cause a 3D model of the physical space to be viewed by another device, either by transmitting the 3D mesh for the other device used to construct a 3D model, or by rendering the 3D model directly on the other device. Other embodiments may be described and/or claimed.

3D mesh generation on a server

Embodiments include systems and methods for creation of a 3D mesh from a video stream or a sequence of frames. A sparse point cloud is first created from the video stream, which is then densified per frame by comparison with spatially proximate frames. A 3D mesh is then created from the densified depth maps, and the mesh is textured by projecting the images from the video stream or sequence of frames onto the mesh. Metric scale of the depth maps may be estimated where direct measurements are not able to be measured or calculated using a machine learning depth estimation network.

3D model reconstruction and scale estimation

Frames for texturing a 3D mesh may be selected to minimize the number of frames required to completely texture the mesh, thus reducing the overhead of texturing. Keyframes are selected from a video stream on the basis of amount of overlap from previously selected keyframes, with the amount of overlap held below a predetermined threshold. The 3D mesh may also be refined and corrected to ensure a higher quality mesh application, including color correction of the selected keyframes. Other embodiments are described.

Efficient texture mapping of a 3-D mesh

Real-time local live texturing of a 3D mesh includes adding keyframes that partially overlap in a sequence of frames to a queue of keyframes. When changes to a property of the 3D mesh created from the sequence of frames meet a predetermined threshold, the face vertices are project into RGB images of the keyframes to test visibility, and the keyframes from which the face is visible is added to a visible keyframe list for each of the faces. A most recently added keyframe from the queue of keyframes is assigned to each face in the 3D mesh, and the 3D mesh is divided into mesh segments based on the assigned keyframes. The keyframe assigned to each of the mesh segments is used to compute texture coordinates for vertices in the respective mesh segment, and an image in the keyframe is assigned as a texture for the respective mesh segment. Colors from the visible keyframe list associated with each of the vertices are averaged into a single RGB value, and the RGB value is assigned to the vertex.

Local live texturing of a 3D mesh

A platform for synchronizing augmented reality (AR) views and information between two or more network connected devices is disclosed. A first device captures a video stream and associated essential meta-data, embeds the essential meta-data into the video stream and transmits it to a second device. The second device receives the video stream, extracts the essential meta-data, inserts one or more AR objects into the video stream with reference to the enhanced meta-data, and transmits to the first device the AR objects and reference to the essential meta-data. The first device renders the one or more AR objects into the video stream, using the essential meta-data references to locate the AR objects in each video stream frame. The second device may also determine and transmit a modified video stream to the first device.

Augmented reality video stream synchronization

Embodiments include systems and methods for remotely measuring distances in an environment captured by a device. A device captures a video stream of a device along with AR data that may include camera pose information and/or depth information, and transmits the video stream and AR data to a remote device. The environment may be analyzed to identify objects such as lines, edges, curves, shapes, anchors/corners, products, e.g., appliances, and other things of interest. Some or all of objects may be identified to the remote device to facilitate selecting an object or region of interest. Selected points for an object may be more precisely located by snapping them to corresponding anchor points for an object. Using anchor points facilitates more precise identification and/or measurement of an aspect of an object, such as one of its dimensions, the volume of a space, or performing other actions such as replacement of an object.

Context aware measurement

Embodiments include systems and methods for generating a 3D mesh from a video stream or other image captured contemporaneously with AR data. The AR data is used to create a depth map, which is then fused with images from frames of the video to form a full 3D mesh. The images and depth map can also be used with an object detection algorithm to recognize 3D objects within the 3D mesh. Methods for fingerprinting the video with AR data captured contemporaneously with each frame are disclosed.

Capture of raw augmented reality data and subsequent 3D mesh generation

Embodiments include systems and methods for filtering augmented reality ('AR') data streams, and in particular, video and/or audio streams to change or remove undesirable content, such as personally identifiable information, embarrassing content, financial data, etc. For example, a handheld device may capture video using a built in camera, and as needed superimpose AR objects on the video to change its content. A cloud service may host an AR session between the handheld device and a remote machine. The cloud service and remote machine may also operate their own filter on the AR data to remove and/or replace content to comply with their interests, regulations, policies, etc. Filtering may be cumulative. For example, the handheld device may remove financial data before it leaves the device, and then the cloud service may replace commercial logos before sharing AR data between session participants.

Identifiable information redaction and/or replacement

Methods and systems disclosed herein are directed to detection and recognition of items of data on labels applied to equipment and identifying metadata labels for the items of data using NLP. Embodiments may include identifying one or more items of data on an image of a label associated with a piece of equipment, determining, using NLP on the one or more items of data of the image, one or more metadata associated, respectively, with the identified one or more items of data, and outputting at least one of the one or more metadata and associated items of data.

Extracting data and metadata from identification labels using natural language processing

Embodiments include systems and methods for determining a 6D pose estimate associated with an image of a physical 3D object captured in a video stream. An initial 6D pose estimate is inferred and then further iteratively refined. The video stream may be frozen to allow the user to tap or touch a display to indicate a location of the user-input keypoints. The resulting 6D pose estimate is used to assist in replacing or superimposing the physical 3D object with digital or virtual content in an augmented reality (AR) frame.

Growing community of inventors

London, United Kingdom

Pavan Kumar Kamaraju