Microsoft Technology Licensing, LLC

Mei-Hsuan Lu

Ming-Chieh Lee

Jian Li

Thomas W Holcomb

Yan Lu

Jiahao Li

Andrey Mikhaylovic Mezentsev

Yongjun Wu

Gary Joseph Sullivan

Chih-Lung Lin

Abhishek Abhishek

Tin Qian

You Zhou

Ryan S Menezes

Chun-Wei Chan

Satya Sasikanth Bendapudi

Bofan Hsu

Pritesh Rajesh Kanani

Binlong Li

Robert Aichner

Siddharth Deepak Mehta

William Edward Miao

Yong Liu

Ermin Kozica

Sivakumar Gobichettypalayam Ananthakrishnan

Rama Krishna Prasad Kasapura Satya Prakash

Arihant Nahata

Diego Villasenor

Rajasekaran Subramanian

Techniques are described for efficiently encoding video data by skipping evaluation of certain encoding modes based on various evaluation criteria. In some solutions, intra-block evaluation is performed in a specific order during encoding, and depending on encoding cost calculations of potential intra-block encoding modes, evaluation of some of the potential modes can be skipped. In some solutions, some encoding modes can be skipped depending on whether blocks are simple (e.g., simple vertical, simple horizontal, or both) or non-simple. In some solutions, various criteria are applied to determine whether chroma-from-luma mode evaluation can be skipped. The various solutions can be used independently and/or in combination.

Determination of coding modes for video content using order of potential coding modes and block classification

Techniques are described for encapsulating AV1 encoded video data within NAL units. For example, the NAL units can be H.264 or HEVC NAL units. Encapsulation can comprise using a reserved NAL unit type. For example, an open bitstream unit comprising AV1 encoded video data can be encapsulated within a NAL unit using a reserved NAL unit type. The NAL unit can be packetized for delivery to another computing device via a computer network.

AV1 codec for real-time video communication

Innovations in hash table construction and hash-based block matching for image encoding or video encoding are described. For example, an encoder determines hash values for base-size candidate blocks in a reference picture. The encoder stores, in a hash table, the hash values for the base-size candidate blocks. The encoder encodes a trial-size current block in a current picture. In some cases, the trial-size current block has a block size larger than the base block size. As part of the encoding, the encoder uses hash-based block matching, between base-size current blocks of the trial-size current block and the base-size candidate blocks, to identify a trial-size matching block, if any, in the reference picture. The encoder stores hash values only for the base-size candidate blocks. This can significantly reduce the computational cost and memory cost for hash table construction during encoding, without hurting compression efficiency or the overall speed of encoding.

Low-cost hash table construction and hash-based block matching for variable-size blocks

Screen content encoding mode evaluation including intra-block evaluation of multiple potential encoding modes

Innovations described herein provide a framework for advertising encoder capabilities, initializing encoder configuration, and signaling run-time control messages for video coding and decoding. For example, an encoding controller receives a request for encoder capability data from a decoding host controller, determines the capability data, and sends the capability data in reply. The capability data can include data that indicate a number of bitstreams, each providing an alternative version of input video, as well as data that indicate scalable video coding capabilities. The decoding host controller creates stream configuration request data based on the encoder capability data, and sends the configuration request data to the encoding controller. During decoding, the decoding host controller can create and send a control message for run-time control of encoding, where the control message includes a stream identifier for a bitstream and layer identifiers for a given layer of the bitstream.

Capability advertisement, configuration and control for video coding and decoding

Technologies for assigning a user-generated issue report to an appropriate team for investigation are disclosed. A log file associated with an issue report is parsed to create an object model. The object model is a hierarchy of objects containing data emitted over time by a computer application associated with the issue report. Rules analyze the content of the object model, and when a rule is satisfied, the issue report is assigned to a team associated with the rule for investigation. Rules may access objects in the object model on a per-component level, thereby allowing issues to be assigned to teams at the level of individual components. Rules may also perform a root cause analysis of the issue based on the object model.

Object model based issue triage

Techniques for recording video from a bitstream are described. In at least some implementations, video data generated as part of a communication session is recorded. According to various implementations, techniques described herein enable portions of an encoded bitstream of video data to be directly recorded as encoded frames and without requiring re-encoding of decoded frames.

Recording video from a bitstream

A user device within a communication architecture, the user device comprising: an image capture device configured to determine image data for the creation of a video channel defining the shared scene; an intrinsic/extrinsic data determiner configured to determine intrinsic/extrinsic capture device data associated with the image capture device; and a video encoder configured to encode the image data and intrinsic/extrinsic capture device data within the video channel.

Shared scene mesh data synchronisation

In various respects, hardware-accelerated decoding is adapted for decoding of video that has been encoded using scalable video coding. For example, for a given picture to be decoded, a host decoder determines whether a corresponding base picture will be stored for use as a reference picture. If so, the host decoder directs decoding with an accelerator such that the some of the same decoding operations can be used for the given picture and the reference base picture. Or, as another example, the host decoder groups encoded data associated with a given layer representation in buffers. The host decoder provides the encoded data for the layer to the accelerator. The host decoder repeats the process layer-after-layer in the order that layers appear in the bitstream, according to a defined call pattern for an acceleration interface, which helps the accelerator determine the layers with which buffers are associated.

Hardware-accelerated decoding of scalable video bitstreams

Innovations described herein provide a generic encoding and decoding framework that includes some features of simulcast and some features of scalable video coding. For example, a bitstream multiplexer multiplexes component bitstreams into a multi-layer encoding (MLE) bitstream that provides temporal scalability, spatial resolution scalability and/or signal to noise ratio scalability. Each of the component bitstreams provides an alternative version of input video, and a given component bitstream can be a non-scalable bitstream or scalable bitstream. The multiplexer follows composition rules for the MLE bitstream and may rewrite values of certain syntax elements of component bitstreams using an approach that avoids bit shifting operations. A corresponding demultiplexer receives an MLE bitstream that includes component bitstreams and demultiplexes at least part of at least one of the component bitstreams from the MLE bitstream, following decomposition rules for the demultiplexing.

Multi-layer encoding and decoding

Ways to mitigate loss in inter-operability scenarios for digital video are presented. For example, a bitstream modification tool (such as a bitstream rewriter running on a network node of a videoconferencing system) receives an incoming bitstream of encoded video (e.g., from an encoder that uses a first loss recovery strategy). The bitstream modification tool processes the incoming bitstream of encoded video to produce an outgoing bitstream of encoded video. In doing so, the bitstream modification tool changes at least one syntax element between the incoming bitstream and the outgoing bitstream so as to mitigate picture loss effects during decoding of the outgoing bitstream under lossy delivery conditions. The bitstream modification tool outputs the outgoing bitstream. In this way, the bitstream modification tool can help avoid blank screens, frozen screens, or other failures during decoding under lossy delivery conditions (e.g., with a decoder that uses a different loss recovery strategy).

Mitigating loss in inter-operability scenarios for digital video

Innovations described herein facilitate the addition of temporal scalability to non-scalable bitstreams. For example, a bitstream rewriter receives units of encoded video data for a non-scalable bitstream from components of a hardware-based encoder. The bitstream rewriter changes at least some of the units of encoded video data so as to produce a scalable bitstream with temporal scalability. In doing so, the bitstream rewriter can associate an original sequence parameter set (SPS) and original picture parameter set (PPS) with pictures for a temporal base layer, and associate a new SPS and new PPS with pictures for a temporal enhancement layer. The bitstream rewriter can also alter syntax elements in the units of encoded video data, for example, changing syntax elements in a slice header in ways that avoid bit shifting operations for following coded slice data for a unit of encoded video data for the temporal enhancement layer.

Adding temporal scalability to a non-scalable bitstream

Optimization of conference call encoding processes is provided. A first client of a multi-party conference call may receive client capability data, including video scalability support, from each of the other clients to the conference call. Based on the client capability data and the transmission capabilities of the first client, including video scalability support, the first client may determine a total number of data streams and properties for each data stream, such that the total number of data streams and the plurality of properties for each data stream are optimized and supported by the respective client capability data and the transmission capabilities. Subsequently, the first client generates one or more data streams according to the total number of data streams and the properties that were determined for each data stream and transmits the one or more data streams to the other clients of the conference call.

Encoding processing for conferencing systems

Mean absolute difference (MAD) prediction for video encoding may be provided. Upon receiving a video stream comprising a plurality of quality layers, a first quantization parameter (QP) may be selected for a first frame of the video stream according to a second QP associated with a second frame and a third QP associated with a third frame. The first frame may then be encoded according to the selected first QP.

Growing community of inventors

Bellevue, WA, United States of America

Mei-Hsuan Lu