Streem, Inc.

Atheer, Inc.

West Texas Technology Partners, LLC

Other

Atheer Labs, Inc.

Ryan R Fink

Sean M Adkinson

Gary Peck

Bryan Ransil

Ryan Phelps

Pavan Kumar Kamaraju

Flora Ponjou Tasse

Ghislain Fouodji Tasse

Nicholas Degroot

Trey Trahin

Alexander Fallenstedt

Brian Gram

Teressa Chizeck

Embodiments include systems and methods for creation of a 3D mesh from a video stream or a sequence of frames. A sparse point cloud is first created from the video stream, which is then densified per frame by comparison with spatially proximate frames. A 3D mesh is then created from the densified depth maps, and the mesh is textured by projecting the images from the video stream or sequence of frames onto the mesh. Metric scale of the depth maps may be estimated where direct measurements are not able to be measured or calculated using a machine learning depth estimation network.

3D model reconstruction and scale estimation

Methods for providing simultaneous access to a hardware data stream to multiple applications are disclosed. The first application to access a hardware device is responsible for providing and publishing an application programming interface (API) that provides access to the hardware device's data stream, which other applications can then call to gain access to the data stream. In some examples, the first application may be a server process or daemon dedicated to managing the hardware device data stream and publishing the API. In some further examples, the first application may instead may carry out user functionality unrelated to managing the hardware device.

Methods and systems for multiple access to a single hardware data stream

Methods and systems for automatic detection and recognition of visual tags on equipment are disclosed. The make and model of an object such as an appliance or consumer device is recognized from an image or video using object detection. This make and model information may then be used to direct a user to locate an equipment information tag that includes model and serial number information. Object recognition and optical character recognition can then be employed to extract the model and serial number from the tag, along with any other relevant information. The extracted information may then be used to locate service and/or operation information.

Automatic detection and recognition of visual tags

Methods for placement of location-persistent 3D objects or annotations in an augmented reality scene are disclosed. By capturing location data along with device spatial orientation and the placement of 3D objects or annotations, the augmented reality scene can be recreated and manipulated. Placed 3D objects or annotations can reappear in a subsequent capture by the same or a different device when brought back to the location of the initial capture and placement of objects. Still further, the placed 3D objects or annotations may be supplemented with additional objects or annotations, or the placed objects or annotations may be removed or modified.

Location persistent augmented reality object and annotation placement

Methods for improving object recognition using depth data are disclosed. An image is captured of a 3-D scene along with depth data, such as in the form of a point cloud. The depth data is correlated with the image of the captured scene, such as by determining the frame of reference of each of the image and the depth data, thereby allowing the depth data to be mapped to the correct corresponding pixels of the image. Object recognition on the image is then improved by employing the correlated depth data. The depth data may be captured contemporaneously with the image of the 3-D scene, such as by using photogrammetry, or at a different time.

Object recognition enhancement using depth data

Switching among sensor feeds for optimum performance includes disposing a processor in communication with sensors, each said sensor providing a primary data stream. A data stream standard is established in the processor. The primary data streams are communicated from the sensors to the processor, and are compared against the data stream in the processor. A secondary data stream is selected from among the primary data streams or synthesized from one or more of the primary data streams, based on which primary data stream(s) most closely match the data stream standard. The secondary data stream is communicated to a data stream recipient. The data stream recipient may identify input in the secondary data stream, and an input executor may execute control commands corresponding to the input so as to control a device or system.

Method and apparatus for switching between sensors

Embodiments include systems and methods for visualizing the position of a capturing device within a 3D mesh, generated from a video stream from the capturing device. A capturing device may provide a video stream along with point cloud data and camera pose data. This video stream, point cloud data, and camera pose data are then used to progressively generate a 3D mesh. The camera pose data and point cloud data can further be used, in conjunction with a SLAM algorithm, to indicate the position and orientation of the capturing device within the generated 3D mesh.

Visualization of camera location in a real-time synchronized 3D mesh

Embodiments include systems and methods for generating a 3D mesh from a video stream or other image captured contemporaneously with AR data. The AR data is used to create a depth map, which is then fused with images from frames of the video to form a full 3D mesh. The images and depth map can also be used with an object detection algorithm to recognize 3D objects within the 3D mesh. Methods for fingerprinting the video with AR data captured contemporaneously with each frame are disclosed.

Capture of raw augmented reality data and subsequent 3D mesh generation

Embodiments include systems and methods for remotely measuring distances in an environment captured by a device. A device captures a video stream of a device along with AR data that may include camera pose information and/or depth information, and transmits the video stream and AR data to a remote device. The remote device receives a selection of a first point and a second point within the video stream and, using the AR data, calculates a distance between the first and second points. The first and second points may be at different locations not simultaneously in view of the device. Other embodiments may capture additional points to compute areas and/or volumes.

Remote measurements from a live video stream

Methods for replacing or obscuring objects detected in an image or video on the basis of image context are disclosed. Context of the image or video may be obtained via pattern recognition on audio associated with the image or video, by user-supplied context, and/or by context derived from image capture, such as the nature of an application used to capture the image. The image or video may be analyzed for object detection and recognition, and depending upon policy, the image or video context used to select objects related or unrelated to the context for replacement or obfuscation. The selected objects may then be replaced with generic objects rendered from 3D models, or blurred or otherwise obscured.

Context-aware selective object replacement

Methods and systems disclosed herein are directed to detection and recognition of items of data on labels applied to equipment and identifying metadata labels for the items of data using NLP. Embodiments may include identifying one or more items of data on an image of a label associated with a piece of equipment, determining, using NLP on the one or more items of data of the image, one or more metadata associated, respectively, with the identified one or more items of data, and outputting at least one of the one or more metadata and associated items of data.

Extracting data and metadata from identification labels using natural language processing

A method for creating and storing a captured image and associated spatial data and augmented reality (AR) data in a file that allows subsequent manipulation and processing of AR objects is disclosed. In embodiments, one or more frames are extracted from a video stream, along with spatial information about the camera capturing the video stream. The one or more frames are analyzed in conjunction with the spatial information to calculate a point cloud of depth data. The one or more frames are stored in a file in a first layer, and the point cloud is stored in the file in a second layer. In some embodiments, one or more AR objects are stored in a third layer.

Layered 3-D images for augmented reality processing

Methods for automatically tagging one or more images and/or video clips using a audio stream are disclosed. The audio stream may be processed using an automatic speech recognition algorithm, to extract possible keywords. The image(s) and/or video clip(s) may then be tagged with the possible keywords. In some embodiments, the image(s) and/or video clip(s) may be tagged automatically. In other embodiments, a user may be presented with a list of possible keywords extracted from the audio stream, from which the user may then select to manually tag the image(s) and/or video clip(s).

Growing community of inventors

Vancouver, WA, United States of America

Ryan R Fink