Apple Inc.

Other

Hsi-Jung Wu

Xiaosong Zhou

Chris Yoochang Chung

Yeping Su

Dazhong Zhang

James O Normile

Alexandros Michael Tourapis

Jiefu Zhai

Thomas Pun

Jae Hoon Kim

David William Singer

Xiaochun Nie

Jun Xin

Hang Yuan

Joe S Abuan

Hyeonkuk Jeong

Douglas Scott Price

Roger Kumar

Yunfei Zheng

Ke-Chiang C Chu

Ke Zhang

Xiaojin Shi

Khaled Mammou

Yan Yang

Fabrice A Robinet

Mei Guo

Peikang Song

Albert E Keinath

Barin Geoffry Haskell

Roberto Garcia, Jr

Ming Chen

Jian Lu

Thomas Christopher Jansen

Yu Tina Tian

Elizabeth Caroline Cranfill

Alican Nalci

Yeqing Wu

Ionut Hristodorescu

Stephen Olivier Lemay

Hao Pan

Yao-Chung Lin

Gobind Johar

Maynard J Handley

Shujie Liu

Sudeng Hu

Yixin Du

Jungsun Kim

Dulce Beatriz Ponceleon

Hilmi Enes Egilmez

Xiaohua Yang

Xin Tong

David R Conrad

Fanyi Duanmu

Jingteng Xue

Xing Wen

Xingyu Zhang

Adriana Dumitras

Christopher M Garrido

Valery G Valentin

Luciano Miguel Verger

Paul S Chang

Atul Puri

Wen Zhao

David L Biderman

Srinivasan Vasudevan

Karthick Santhanam

John Y Su

Katherine Shu-Wei Wang

Yuxin Liu

Jun Xu

James David Batson

Roberto Garcia

Haitao Guo

Ming Jin

Krishnakanth Rapaka

Jim Normile

Gianluca Filippini

Christian L Duvivier

Feng Yi

Zhenheng Li

Guoxin Jin

Guy Cote

Beibei Wang

Jim Chen Chou

Ting Chen

Sorin Constantin Cismas

Berkat S Tung

Christopher L Flick

Patrick Miauton

Ian J Baird

David Daming Kuo

Eric L Chien

Samir Gehani

Andrew Yanowitz

Sundararaman V Shiva

Vikrant Kasarabada

Kevin Arthur Robertson

Johnny Trenh

Hassan A Shojania

Gurtej Singh Chandok

An encoder or decoder can perform enhanced motion vector prediction by receiving an input block of data for encoding or decoding and accessing stored motion information for at least one other block of data. Based on the stored motion information, the encoder or decoder can generate a list of one or more motion vector predictor candidates for the input block in accordance with an adaptive list construction order. The encoder or decoder can predict a motion vector for the input block based on at least one of the one or more motion vector predictor candidates.

Enhanced motion vector prediction

Techniques are proposed to improve temporal motion projection in video coding. Candidate reference frames available for use in temporal motion projection are sorted in processing order according to scores assigned based on estimates of the reference frames' suitability for prediction. Such estimates may be based on temporal distance between each candidate reference frame and that reference frame's prediction references. Estimates may be based, for each reference frame, based on an estimate of coding quality of a reference frame from which the respective candidate reference frame makes a prediction reference. Once sorted, the candidate reference frames may be processing in the sorting order to supply prediction data to a current frame that is to be coded from the candidate reference frames. Additionally, hardware friendly designs of motion field hole filling and motion vector smoothing operations are proposed. Such designs can reduce hardware implementation complexity and benefit hardware parallel processing in several aspects: by removing the dependency among different processing block rows for hole filling and motion vector smoothing so that it is becomes easier and friendlier to achieve hardware parallel processing; by reducing the hardware bandwidth loading overhead; by improving hardware pipeline throughput; and/or by avoiding adding a line buffer to store the data from the above row since a line buffer will increase hardware cost.

Hardware friendly design for motion field processing and quality improvement of motion field

Improved neural-network-based image and video coding techniques are presented, including hybrid techniques that include both tools of a host codec and neural-network-based tools. In these improved techniques, the host coding tools may include conventional video coding standards such H.266 (VVC). In an aspects, source frames may be partitioned and either host or neural-network-based tools may be selected per partition. Coding parameter decisions for a partition may be constrained based on the partitioning and coding tool selection. Rate control for host and neural network tools may be combined. Multi-stage processing of neural network output may use a checkerboard prediction pattern.

Hybrid neural network based end-to-end image and video coding method

Techniques are described for express and implied signaling of transform mode selections in video coding. Information derived from coefficient samples in a given transform unit (TU) or prediction unit (PU) may constrain or modify signaling of certain syntax elements at the coding block (CB), TU, or PU levels. For instance, based on the spatial locations of decoded coefficients, the spatial patterns of coefficients, or the correlation with the coefficients in neighboring blocks, various syntax elements such as the transform type and related flags/indices or secondary transform modes/flags indices, a residual coding mode, intra and inter prediction modes, and scanning order may be disabled or constrained. In another case, if the coefficient samples match a desired spatial pattern or have other desired properties then a default transform type, a default secondary transform type, a default intra and inter prediction mode or other block level modes may be inferred at the decoder side. Similar ideas may be extended to infer intra-block copy (IBC) flags/modes/indices from coefficient samples, or to infer flags and modes related to predictive coding techniques such as block-wise differential pulse code modulation (BDPCM), i.e., horizontal and vertical BDPCM modes, palette mode syntax and other block level flags/indices.

Coefficient-based transform and mode signaling

In an example method, a decoder obtains a data stream representing video content. The video content is partitioned into one or more logical units, and each of the logical units is partitioned into one or more respective logical sub-units. The decoder determines that the data stream includes first data indicating that a first logical unit has been encoded according to a flexible skip coding scheme. In response, the decoder determines a first set of decoding parameters based on the first data, and decodes each of the logical sub-units of the first logical unit according to the first set of decoding parameters.

Encoding and decoding video content using prediction-aware flexible skip coding

A cross-component based filtering system is disclosed for video coders and decoders. The filtering system may include a filter having an input for a filter offset and an input for samples reconstructed from coded video data representing a native component of source video on which the filter operates. The offset may be generated at least in part from a sample classifier that classifies samples reconstructed from coded video data representing a color component of the source video orthogonal to the native component according to sample intensity.

Enhanced cross component sample offset filtering system for video coding and decoding

Techniques are disclosed for generating virtual reference frames that may be used for prediction of input video frames. The virtual reference frames may be derived from already-coded reference frames and thereby incur reduced signaling overhead. Moreover, signaling of virtual reference frames may be avoided until an encoder selects the virtual reference frame as a prediction reference for a current frame. In this manner, the techniques proposed herein contribute to improved coding efficiencies.

Predictive video coding employing virtual reference frames generated by direct MV projection (DMVP)

Video compression and decompression techniques are disclosed that provide improved bandwidth control for video compression and decompression systems. In particular, video coding and decoding techniques quantize input video in multiple dimensions. According to these techniques, pixel residuals may be generated from a comparison of an array of input data to an array of prediction data. The pixel residuals may be quantized in a first dimension. After the quantization, the quantized pixel residuals may be transformed to an array of transform coefficients. The transform coefficients may be quantized in a second dimension and entropy coded. Decoding techniques invert these processes. In still other embodiments, multiple quantizers may be provided upstream of the transform stage, either in parallel or in cascade, which provide greater flexibility to video coders to quantize data in different dimensions in an effort to balance the competing interest in compression efficiency and quality of reconstructed video.

Multidimensional quantization techniques for video coding/decoding systems

Some embodiments provide a method for initiating a video conference using a first mobile device. The method presents, during an audio call through a wireless communication network with a second device, a selectable user-interface (UI) item on the first mobile device for switching from the audio call to the video conference. The method receives a selection of the selectable UI item. The method initiates the video conference without terminating the audio call. The method terminates the audio call before allowing the first and second devices to present audio and video data exchanged through the video conference.

Establishing a video conference during a phone call

Video coders and decoders perform transform coding and decoding on blocks of video content according to an adaptively selected transform type. The transform types are organized into a hierarchy of transform sets where each transform set includes a respective number of transforms and each higher-level transform set includes the transforms of each lower-level transform set within the hierarchy. The video coders and video decoders may exchange signaling that establishes a transform set context from which a transform set that was selected for coding given block(s) may be identified. The video coders and video decoders may exchange signaling that establishes a transform decoding context from which a transform that was selected from the identified transform set to be used for decoding the transform unit. The block(s) may be coded and decoded by the selected transform.

Incremental transform set construction and transform signaling for video coding

Techniques are disclosed for deriving prediction pixel blocks for use in intra-coding video and combined inter- and intra-coding video. In a first aspect, the techniques may include deriving value(s) for pixel location(s) of the prediction pixel block by, when a prediction direction vector assigned to the prediction vector points to quadrants I or III of a Cartesian plane, deriving the pixel location's value from pixel values in two regions of previously-decoded pixel data intercepted by extending the prediction direction vector in two opposite directions through the pixel location. When the prediction direction vector points toward quadrants II of the Cartesian plane, deriving the pixel location's value from pixel values in one region intercepted by the prediction direction vector through the pixel location, and from a second region intercepted by a vector that is orthogonal to the prediction direction vector.

Smoothed directional and DC intra prediction

Systems and methods disclosed for video compression, utilizing neural networks for predictive video coding. Processes employed combine multiple banks of neural networks with codec system components to carry out the coding and decoding of video data.

Neural network based residual coding and prediction for predictive coding

Techniques are disclosed by which a coding parameter is determined to encode video data resulting in encoded video data possessing a highest possible video quality. Features may be extracted from an input video sequence. The extracted features may be compared to features described in a model of coding parameters generated by a machine learning algorithm from reviews of previously-coded videos, extracted features of the previously-coded videos, and coding parameters of the previously-coded videos. When a match is detected between the extracted features of the input video sequence and extracted features represented in the model, a determination may be made as to whether coding parameters that correspond to the matching extracted feature correspond to a tier of service to which the input video sequence is to be coded. When the coding parameters that correspond to the matching extracted feature correspond to the tier of service to which the input video sequence is to be coded, the input video sequence may be coded according to the coding parameters.

Content adaptation for streaming

A system obtains a data set representing immersive video content for display at a display time, including first data representing the content according to a first level of detail, and second data representing the content according to a second higher level of detail. During one or more first times prior to the display time, the system causes at least a portion of the first data to be stored in a buffer. During one or more second times prior to the display time, the system generates a prediction of a viewport for displaying the content to a user at the display time, identifies a portion of the second data corresponding to the prediction of the viewport, and causes the identified portion of the second data to be stored in the video buffer. At the display time, the system causes the content to be displayed to the user using the video buffer.

Immersive video streaming using view-adaptive prefetching and buffer control

In communication applications, aggregate source image data at a transmitter exceeds the data that is needed to display a rendering of a viewport at a receiver. Improved streaming techniques that include estimating a location of a viewport at a future time. According to such techniques, the viewport may represent a portion of an image from a multi-directional video to be displayed at the future time, and tile(s) of the image may be identified in which the viewport is estimated to be located. In these techniques, the image data of tile(s) in which the viewport is estimated to be located may be requested at a first service tier, and the other tile in which the viewport is not estimated to be located may be requested at a second service tier, lower than the first service tier.

Adaptive coding and streaming of multi-directional video

Growing community of inventors

San Jose, CA, United States of America

Hsi-Jung Wu