Apple Inc.

Thomson Licensing

Google Inc.

Dolby Laboratories Licensing Corporation

Meta Platforms, Inc.

Dolby International Ab

Jae Hoon Kim

Hsi-Jung Wu

Xiaosong Zhou

Dazhong Zhang

Chris Yoochang Chung

Jiefu Zhai

Hang Yuan

Peikang Song

Ming Chen

Yeping Su

Yunfei Zheng

Sudeng Hu

Xing Wen

Alexandros Michael Tourapis

Cristina Gomila

David R Conrad

Peng Yin

Ajay K Luthra

Arjun Ramamurthy

Michael Andrew Grossman

Xiaohua Yang

Jingteng Xue

Shankar Lakshmi Regunathan

Christopher M Garrido

Karthick Santhanam

Ming Jin

Xue Hai Fang

Patrick Miauton

Albert E Keinath

Wei-Ying Kung

Cosmin Vasile Stejerean

Pankaj Sethi

Christina Gomila

In communication applications, aggregate source image data at a transmitter exceeds the data that is needed to display a rendering of a viewport at a receiver. Improved streaming techniques that include estimating a location of a viewport at a future time. According to such techniques, the viewport may represent a portion of an image from a multi-directional video to be displayed at the future time, and tile(s) of the image may be identified in which the viewport is estimated to be located. In these techniques, the image data of tile(s) in which the viewport is estimated to be located may be requested at a first service tier, and the other tile in which the viewport is not estimated to be located may be requested at a second service tier, lower than the first service tier.

Adaptive coding and streaming of multi-directional video

Techniques are disclosed for coding video data predictively based on predictions made from spherical-domain projections of input pictures to be coded and reference pictures that are prediction candidates. Spherical projection of an input picture and the candidate reference pictures may be generated. Thereafter, a search may be conducted for a match between the spherical-domain representation of a pixel block to be coded and a spherical-domain representation of the reference picture. On a match, an offset may be determined between the spherical-domain representation of the pixel block to a matching portion of the of the reference picture in the spherical-domain representation. The spherical-domain offset may be transformed to a motion vector in a source-domain representation of the input picture, and the pixel block may be coded predictively with reference to a source-domain representation of the matching portion of the reference picture.

Sphere projected motion estimation/compensation and mode decision

Embodiments of the present disclosure provide systems and methods for perspective shifting in a video conferencing session. In one exemplary method, a video stream may be generated. A foreground element may be identified in a frame of the video stream and distinguished from a background element of the frame. Data may be received representing a viewing condition at a terminal that will display the generated video stream. The frame of the video stream may be modified based on the received data to shift of the foreground element relative to the background element. The modified video stream may be displayed at the displaying terminal.

Systems and methods for perspective shifting in video conferencing session

A video to be encoded to a plurality of different target encodings for bandwidth adaptive serving is received. The video is encoded into a plurality of different candidate encodings using different candidate encoding parameters. A quality metric is determined for each of the plurality of different candidate encodings. One or more different target quality metrics are selected for a first portion of the different target encodings based at least in part on one or more specified constraints for one or more target devices. One or more different target quality metrics are selected for a second portion of the different target encodings based at least in part on the determined quality metrics of the different candidate encodings. Based at least in part on the selected different target quality metrics for the first portion and the second portion, the plurality of different target encodings of the video is generated.

Constraint-modified selection of video encoding configurations

Techniques are disclosed for coding and decoding video data using object recognition and object modeling as a basis of coding and error recovery. A video decoder may decode coded video data received from a channel. The video decoder may perform object recognition on decoded video data obtained therefrom, and, when an object is recognized in the decoded video data, the video decoder may generate a model representing the recognized object. It may store data representing the model locally. The video decoder may communicate the model data to an encoder, which may form a basis of error mitigation and recovery. The video decoder also may monitor deviation patterns in the object model and associated patterns in audio content; if/when video decoding is suspended due to operational errors, the video decoder may generate simulated video data by analyzing audio data received during the suspension period and developing video data from the data model and deviation(s) associated with patterns detected from the audio data.

Applications for decoder-side modeling of objects identified in decoded video data

Techniques are presented for modifying images of an object in video, for example to correct for lens distortion, or to beautify a face. These techniques include extracting and validating features of an object from a source video frame, tracking those features over time, estimating a pose of the object, modifying a 3D model of the object based on the features, and rendering a modified video frame based on the modified 3D model and modified intrinsic and extrinsic matrices. These techniques may be applied in real-time to an object in a sequence of video frames.

Real-time face and object manipulation

Methods and Systems disclosed to counteract spatial distortions introduced by imaging processes of multi-directional video frames, where objects may be projected to spherical or equirectangular representations. Techniques provided to invert the spatial distortions in video frames used as reference picture data in predictive coding, by spatially transforming the image content of the reference picture data before this image content is being used for the prediction of input video data in prediction-based coders and decoders.

Processing of equirectangular object data to compensate for distortion by spherical projections

Systems and methods for coding a video to be overlaid by annotations are devised. A motion compensated predictive coding is employed, wherein coding parameters of video pixel blocks are determined based on the pixel blocks' relation to the annotations. A decoder decodes the video and annotates it based on metadata, obtained from the coder or other sources, describing the annotations' appearance and rendering mode.

Efficient coding of video data in the presence of video annotations

Chroma deblock filtering of reconstructed video samples may be performed to remove blockiness artifacts and reduce color artifacts without over-smoothing. In a first method, chroma deblocking may be performed for boundary samples of a smallest transform size, regardless of partitions and coding modes. In a second method, chroma deblocking may be performed when a boundary strength is greater than 0. In a third method, chroma deblocking may be performed regardless of boundary strengths. In a fourth method, the type of chroma deblocking to be performed may be signaled in a slice header by a flag. Furthermore, luma deblock filtering techniques may be applied to chroma deblock filtering.

In loop chroma deblocking filter

Techniques are disclosed for managing display of content from multi-view video data. According to these techniques, an object may be identified from content of the multi-view video. The object's location may be tracked across a sequence of multi-view video. The technique may extract a sub-set of video that is contained within a view window that is shifted in an image space of the multi-view video in correspondence to the tracked object's location. These techniques may be implemented either in an image source device or an image sink device.

Object tracking in multi-view video

Techniques are disclosed for managing memory allocations when coding video data according to multiple codec configurations. According to these techniques, devices may negotiate parameters of a coding session that include parameters of a plurality of different codec configurations that may be used during the coding session. A device may estimate sizes of decoded picture buffers for each of the negotiated codec configurations and allocate in its memory a portion of memory sized according to a largest size of the estimated decoded picture buffers. Thereafter, the devices may exchange coded video data. The exchange may involve decoding coded data of reference pictures and storing the decoded reference pictures in the allocated memory. During the coding session, the devices may toggle among the different negotiated codec configurations. As they do, reallocations of memory may be avoided.

Dynamic video configurations

A method () and apparatus () for three-dimensional television (3DTV) image adjustment includes loading () default 2D-to-3D image setting values from a default settings memory to a user adjustment settings memory, annunciating () the default 2D-to-3D image setting values, receiving () a 2D-to-3D image settings value adjustment, saving () the 2D-to-3D image settings value adjustment in the user adjustment settings memory, and applying () the 2D-to-3D image settings value adjustment to a 2D-to-3D converted image. These methods and apparatuses allow individual users to set 3DTV image settings to their personal preferences to compensate for brightness reductions caused by 3DTV glasses, depth perception sensitivities, and other image quality factors.

Methods and apparatus for 3DTV image adjustment

Aspects of the present disclosure provide techniques for reducing latency and improving image quality of a viewport extracted from multi-directional video communications. According to such techniques, first streams of coded video data are received from a source. The first streams include coded data for each of a plurality of tiles representing a multi-directional video, where each tile corresponding to a predetermined spatial region of the multi-directional video, and at least one tile of the plurality of tiles in the first streams contains a current viewport location at a receiver. The techniques include decoding the first streams and displaying the tile containing the current viewport location. When the viewport location at the receiver changes to include a new tile of the plurality of tiles, retrieving and decoding first streams for the new tile, displaying the decoded content for the changed viewport location, and transmitting the changed viewport location to the source.

Scalability of multi-directional video streaming

Growing community of inventors

San Jose, CA, United States of America

Jae Hoon Kim