Microsoft Technology Licensing, LLC

Other

Microsoft Corporation

Ming-Chieh Lee

Chih-Lung Lin

Wei-ge Chen

Naveen Thumpudi

Thomas W Holcomb

You Zhou

Pohsiang Hsu

Mei-Hsuan Lu

Wei-Ge Chen

Sridhar Srinivasan

Sergey Sablin

Chuang Gu

Gary Joseph Sullivan

Jian Li

Yan Lu

Jiahao Li

Sriram Srinivasan

David Yuheng Zhao

Timothy Mark Moore

Pontus Carlsson

Mattias Nilsson

Mu Han

Stacey Spears

Xiping Zuo

Yongjun Wu

Shankar Lakshmi Regunathan

Abhishek Abhishek

Tin Qian

Ryan S Menezes

Brian D Crites

David Gray

Jie Liang

Tim M Moore

Sergey Silkin

Albert Szu-Chi Wang

Chih-Lung (Bruce) Lin

Andrey Mikhaylovic Mezentsev

Binlong Li

Sergei Nikiforov

Magnus Hemmendorff

Philip Lawrence Garrett

Satya Sasikanth Bendapudi

Amy Jennifer Beauford

Anil Murching

Amy Lu

Jordi Ribas-Corbera

Cheng Chang

William Chambers Powell, Iii

Ce Wang

Bruce Chih-Lung Lin

Chun-Wei Chan

Sergey Smirnov

Bofan Hsu

Robert Aichner

Jizhe Jin

Walid S I Ali

Shou-Jen Wu

Ermin Kozica

Shou-Jen (Joseph) Wu

Shao-Ting Wang

Tom W Holcomb

Chih-lung Bruce Lin

Mattias Dan Nilsson

Apparatus and methods are disclosed for using machine learning models with private and public domains. Operations can be applied to transform input to a machine learning model in a private domain that is kept secret or otherwise made unavailable to third parties. In one example of the disclosed technology, a method includes applying a private transform to produce transformed input, providing the transformed input to a machine learning model that was trained using a training set modified by the private transform, and generating inferences with the machine learning model using the transformed input. Examples of suitable transforms that can be employed include matrix multiplication, time or spatial domain to frequency domains, and partitioning a neural network model such that an input and at least one hidden layer form part of the private domain, while the remaining layers form part of the public domain.

Protecting deep learned models

A format for use in encoding moving image data, comprising: a sequence of frames including plurality of the frames in which at least a region is encoded using motion estimation; a respective set of motion vector values representing motion vectors of the motion estimation for each respective one of these frames or each respective one of one or more regions within each of such frames; and at least one respective indicator associated with each of the respective frames or regions, indicating whether the respective motion vector values of the respective frame or region are encoded at a first resolution or a second resolution.

Representing motion vectors in an encoded bitstream

Implementations of the subject matter described herein provide a solution for rate control based on reinforcement learning. In this solution, an encoding state of a video encoder is determined, the encoding state being associated with encoding of a first video unit by the video encoder. An encoding parameter associated with rate control in the video encoder is determined by a reinforcement learning model and based on the encoding state of the video encoder. A second video unit different from the first video unit is encoded based on the encoding parameter. In this way, it is possible to achieve a better quality of experience (QOE) for real time communication with computation overhead being reduced.

Reinforcement learning based rate control

When encoding/decoding a current block of a current picture using intra block copy ('BC') prediction, the location of a reference block is constrained so that it can be entirely within an inner search area of the current picture or entirely within an outer search area of the current picture, but cannot overlap both the inner search area and the outer search area. In some hardware-based implementations, on-chip memory buffers sample values of the inner search area, and off-chip memory buffers sample values of the outer search area. By enforcing this constraint on the location of the reference block, an encoder/decoder can avoid memory access operations that are split between on-chip memory and off-chip memory when retrieving the sample values of the reference block. At the same time, a reference block close to the current block may be used for intra BC prediction, helping compression efficiency.

Constraints on locations of reference blocks for intra block copy prediction

Innovations in adaptive encoding of screen content based on motion type are described. For example, a video encoder system receives a current picture of a video sequence. The video encoder system determines a current motion type for the video sequence and, based at least in part on the current motion type, sets one or more encoding parameters. Then, the video encoder system encodes the current picture according to the encoding parameter(s). The innovations can be used in real-time encoding scenarios when encoding screen content for a screen sharing application, desktop conferencing application, or other application. In some cases, the innovations allow a video encoder system to adapt compression to different characteristics of screen content at different times within the same video sequence.

Adaptive encoding of screen content based on motion type

Techniques are described for efficiently encoding video data by skipping evaluation of certain encoding modes based on various evaluation criteria. In some solutions, intra-block evaluation is performed in a specific order during encoding, and depending on encoding cost calculations of potential intra-block encoding modes, evaluation of some of the potential modes can be skipped. In some solutions, some encoding modes can be skipped depending on whether blocks are simple (e.g., simple vertical, simple horizontal, or both) or non-simple. In some solutions, various criteria are applied to determine whether chroma-from-luma mode evaluation can be skipped. The various solutions can be used independently and/or in combination.

Determination of coding modes for video content using order of potential coding modes and block classification

Techniques are described for encapsulating AV1 encoded video data within NAL units. For example, the NAL units can be H.264 or HEVC NAL units. Encapsulation can comprise using a reserved NAL unit type. For example, an open bitstream unit comprising AV1 encoded video data can be encapsulated within a NAL unit using a reserved NAL unit type. The NAL unit can be packetized for delivery to another computing device via a computer network.

AV1 codec for real-time video communication

Innovations in hash table construction and hash-based block matching for image encoding or video encoding are described. For example, an encoder determines hash values for base-size candidate blocks in a reference picture. The encoder stores, in a hash table, the hash values for the base-size candidate blocks. The encoder encodes a trial-size current block in a current picture. In some cases, the trial-size current block has a block size larger than the base block size. As part of the encoding, the encoder uses hash-based block matching, between base-size current blocks of the trial-size current block and the base-size candidate blocks, to identify a trial-size matching block, if any, in the reference picture. The encoder stores hash values only for the base-size candidate blocks. This can significantly reduce the computational cost and memory cost for hash table construction during encoding, without hurting compression efficiency or the overall speed of encoding.

Low-cost hash table construction and hash-based block matching for variable-size blocks

Screen content encoding mode evaluation including intra-block evaluation of multiple potential encoding modes

Innovations described herein provide a framework for advertising encoder capabilities, initializing encoder configuration, and signaling run-time control messages for video coding and decoding. For example, an encoding controller receives a request for encoder capability data from a decoding host controller, determines the capability data, and sends the capability data in reply. The capability data can include data that indicate a number of bitstreams, each providing an alternative version of input video, as well as data that indicate scalable video coding capabilities. The decoding host controller creates stream configuration request data based on the encoder capability data, and sends the configuration request data to the encoding controller. During decoding, the decoding host controller can create and send a control message for run-time control of encoding, where the control message includes a stream identifier for a bitstream and layer identifiers for a given layer of the bitstream.

Capability advertisement, configuration and control for video coding and decoding

Techniques and tools for video coding/decoding with motion resolution switching and sub-block transform coding/decoding are described. For example, a video encoder adaptively switches the resolution of motion estimation and compensation between quarter-pixel and half-pixel resolutions; a corresponding video decoder adaptively switches the resolution of motion compensation between quarter-pixel and half-pixel resolutions. For sub-block transform sizes, for example, a video encoder adaptively switches between 8&#xd7;8, 8&#xd7;4, and 4&#xd7;8 DCTs when encoding 8&#xd7;8 prediction residual blocks; a corresponding video decoder switches between 8&#xd7;8, 8&#xd7;4, and 4&#xd7;8 inverse DCTs during decoding.

Video coding/decoding with sub-block transform sizes and adaptive deblock filtering

Growing community of inventors

Bellevue, WA, United States of America

Ming-Chieh Lee