Microsoft Technology Licensing, LLC

Facebook, Inc.

Meta Platforms, Inc.

Shyam Sadhwani

Yongjun Wu

Naveen Thumpudi

Bakkama Srinath Reddy

Gary Joseph Sullivan

Glenn Frederick Evans

Matthew Richard Wozniak

Lihua Zhu

Sudhakar Visweswara Prabhu

Gang Ji

Sandeep Kanumuri

Weidong Zhao

Wenfeng Gao

Nikola Borisov

Vladislav Alexandrov

David H Mebane

Dibyajyoti Roy

B Anil Kumar

Stephen Estrop

Wenbo Zhang

Victor Cherepanov

Sridhar Sankuratri

Brian Patrick Evans

Eric Christoffersen

Donald J Munsil

Haoyun Wu

Yuechuan Li

Anand Ganesh

Chi Wang Ho

Samuel John Wenker

Hany Farag

Robert K Hughes

Arthur Cavalcanti Alem

Bret Lorimore

Shijun Sun

Thomas W Holcomb

Costin Hagiu

Ziyad Ibrahim

Shir Aharon

Firoz Dalal

Saswata Mandal

Jeroen E Van Eesteren

Florin Folta

Lucia Darsa

Yu-Chen Sun

Aaron Oneal

Carol Greenbaum

Abo Talib Mahfoodh

Daniele Terdina

Fedir Yuriyovych Kyslov

Yun Zhang

Balachandar Sivakumar

Methods, systems, and storage media to determine an available number of hardware decoders that can be used to process a video feed. The system can receive a select group of participants from a plurality of participants associated with the video call. The system can include identify a video feed for each participant of the select group of participants. The system can send a request for the video feed of each participant of the select group of participants. The system can receive a video feed associated with each of the select group of participants in the video call. The system can allocate a hardware decoder to each participant of the select group based on the request for each video feed. The system can determine operational parameters of at least one of the hardware decoders. The system can include switching the video feed from the hardware decoder to the software decoder.

Systems and methods for video decoder management

A media processing tool adds custom data to an elementary media bitstream or media container. The custom data indicates nominal range of samples of media content, but the meaning of the custom data is not defined in the codec format or media container format. For example, the custom data indicates the nominal range is full range or limited range. For playback, a media processing tool parses the custom data and determines an indication of media content type. A rendering engine performs color conversion operations whose logic changes based at least in part on the media content type. In this way, a codec format or media container format can in effect be extended to support full nominal range media content as well as limited nominal range media content, and hence preserve full or correct color fidelity, while maintaining backward compatibility and conformance with the codec format or media container format.

Custom data indicating nominal range of samples of media content

Video image stabilization provides better performance on a generic platform for computing devices by evaluating available multimedia digital signal processing components, and selecting the available components to utilize according to a hierarchy structure for video stabilization performance for processing parts of the video stabilization. The video stabilization has improved motion vector estimation that employs refinement motion vector searching according to a pyramid block structure relationship starting from a downsampled resolution version of the video frames. The video stabilization also improves global motion transform estimation by performing a random sample consensus approach for processing the local motion vectors, and selection criteria for motion vector reliability. The video stabilization achieves the removal of hand shakiness smoothly by real-time one-pass or off-line two-pass temporal smoothing with error detection and correction.

Motion vector estimation for video image stabilization

Approaches to selectively using start code emulation prevention ('SCEP') on encoded data for media content are described herein. For example, a media encoder selectively performs SCEP processing on encoded data for media content, and sets a value of a syntax element that indicates whether or not to perform SCEP processing on the encoded data. The encoder stores the encoded data for output as part of a bitstream, where the syntax element is signaled in association with the bitstream. A media decoder receives the encoded data, determines, from the value of the syntax element, whether or not to perform SCEP processing on the encoded data, and selectively performs SCEP processing on the encoded data. In this way, the computational cost of scanning operations for SCEP processing can be avoided in many scenarios, and bit rate increases due to insertion of SCEP bytes can be limited.

Selective use of start code emulation prevention

Systems, methods, and non-transitory computer-readable media can identify a set of participants in a group video call, wherein each participant is associated with an uplink capacity and a downlink capacity, and the set of participants includes a set of sender participants and a set of subscriber participants. For a first sender participant of the set of sender participants, one or more video stream layers to be uploaded by the first sender participant are determined based on downlink capacities of one or more subscriber participants of the set of subscriber participants. Each subscriber participant of the one or more subscriber participants is assigned to receive one video stream layer of the one or more video stream layers to be uploaded by the first sender participant.

Systems and methods for optimizing simulcast streams in group video calls

Syntax structures that indicate the completion of coded regions of pictures are described. For example, a syntax structure in an elementary bitstream indicates the completion of a coded region of a picture. The syntax structure can be a type of network abstraction layer unit, a type of supplemental enhancement information message or another syntax structure. For example, a media processing tool such as an encoder can detect completion of a coded region of a picture, then output, in a predefined order in an elementary bitstream, syntax structure(s) that contain the coded region as well as a different syntax structure that indicates the completion of the coded region. Another media processing tool such as a decoder can receive, in a predefined order in an elementary bitstream, syntax structure(s) that contain a coded region of a picture as well as a different syntax structure that indicates the completion of the coded region.

Syntax structures indicating completion of coded regions

Innovations in reconstruction and rendering of panoramic video are described. For example, a view-dependent operation controller of a panoramic video playback system receives an indication of a view direction for an application and, based at least in part on the view direction, identifies a section of a picture of panoramic video in an input projection. The view-dependent operation controller limits operations of a color converter, video decoder, and/or streaming controller to the identified section. In this way, the panoramic video playback system can avoid performing operations to reconstruct sections of the picture of panoramic video that will not be viewed. As another example, a mapper of a panoramic video playback system re-projects at least some sample values in an input flat projection towards a center location for a view direction, producing an output flat projection, which an application can use to generate one or more screen projections.

Re-projecting flat projections of pictures of panoramic video for rendering by application

Video frames of a higher-resolution chroma sampling format such as YUV 4:4:4 are packed into video frames of a lower-resolution chroma sampling format such as YUV 4:2:0 for purposes of video encoding. For example, sample values for a frame in YUV 4:4:4 format are packed into two frames in YUV 4:2:0 format. After decoding, the video frames of the lower-resolution chroma sampling format can be unpacked to reconstruct the video frames of the higher-resolution chroma sampling format. In this way, available encoders and decoders operating at the lower-resolution chroma sampling format can be used, while still retaining higher resolution chroma information. In example implementations, frames in YUV 4:4:4 format are packed into frames in YUV 4:2:0 format such that geometric correspondence is maintained between Y, U and V components for the frames in YUV 4:2:0 format.

Frame packing and unpacking higher-resolution chroma sampling formats

Video content is protected using a digital rights management (DRM) mechanism, the video content having been previously encrypted and compressed for distribution, and also including metadata such as closed captioning data, which might be encrypted or clear. The video content is obtained by a system of a computing device, the metadata is extracted from the video content and provided to a video decoder, and the video content is provided to a secure DRM component. The secure DRM component decrypts the video content and provides the decrypted video content to a secure decoder component of a video decoder. As part of the decryption, the secure DRM component drops the metadata that was included in the obtained video content. However, the video decoder receives the extracted metadata in a non-protected environment and thus is able to provide the extracted metadata and the decoded video content to a content playback application.

Protected media decoding system supporting metadata

Techniques are described for communicating encoded data using start code emulation prevention. The described techniques include obtaining at least one partially encrypted packet, identifying at least one portion of the packet that is unencrypted, and determining that the identified unencrypted portion(s) emulates a start code. Start code emulation prevention data or emulation prevention bytes (EPBs) may be inserted into only the encrypted portion of the packet. The modified packet may be communicated to another device/storage, along with an indication of which portion(s) of the packet are unencrypted. Upon receiving the packet and indication, the receiving device may identify and remove the EPBs in the identified unencrypted portion(s) of the packet, and decrypt the packet to recover the data. In some aspects, upon identifying the indication, the receiving device may only search for EPBs in the unencrypted portion(s) of the packet, thus yielding a more efficient start code emulation prevention process.

MPEG transport frame synchronization

Innovations in opportunistic frame dropping for variable-frame-rate encoding of digital video are presented. In general, a computing system selectively drops a frame when the cost of encoding the frame (e.g., in terms of use of computational resources and/or power) is expected to outweigh the benefit of encoding the frame (e.g., in terms of better quality). For example, a frame dropping module detects whether there is significant change in a given frame relative to a control frame, which is a previous frame stored in a control frame buffer. If significant change is detected, the frame dropping module stores the given frame in the control frame buffer, thereby replacing the control frame, and passes the given frame to a video encoder. Otherwise, the frame dropping module drops the given frame without replacing the control frame in the control frame buffer and without passing the given frame to the video encoder.

Opportunistic frame dropping for variable-frame-rate encoding

Growing community of inventors

Bellevue, WA, United States of America

Shyam Sadhwani