Scene encoding for multi-user mixed-reality telepresence

Abstract

A scene encoding subsystem for multi-user mixed-reality telepresence has at least one encoder that processes signals from a plurality of cameras. A scene manager determines relative priorities of the cameras and allocates CPU resources and GPU resources in the at least one encoder based on the relative priorities of the cameras, where (i) greater CPU resources are allocated to higher priority cameras than to lower priority cameras and (ii) greater GPU resources are allocated to lower priority cameras than to higher priority cameras. The scene manager instructs a CPU to process a skipped frame using a bounding-box expansion algorithm based on object motion determined using GPU object detection and segmentation processing of previous non-skipped frames. The scene manager multiplicatively decreases and additively increases CPU and GPU resources for a set of multiple cameras based on performance of any one camera in the set.