Real-time system for streaming with live volumetric capture and associated metadata

Abstract

Disclosed are systems and methods that leverage machine learning techniques to provide information about items that are on display during a live event. A 3D volumetric capture system is used to construct a 3D representation of the event in real time. For example, depth and color data from multiple perspectives may be used to construct a 3D mesh of colored polygons that represent the scene. Users may view the live event with a 3D enabled virtual reality or augmented reality headset. Background subtraction may be used to isolate items of interest. In some configurations a machine learning model may process the 3D mesh to generate a textual description of items in the scene. The textual description may then be used to search for more information about a particular item. Additionally, or alternatively, the machine learning model may directly predict one or more items in the scene.