Koninklijke Kpn N.v.

Nederlandse Organisatie Voor Toegepast-Natuurwetenschappelijk Onderzoek Tno

Other

Koninklike Kpn N.v.

Emmanuel Thomas

Omar Aziz Niamut

Ray Van Brandenburg

Mattijs Oskar Van Deventer

Hans Maarten Stokking

Martin Prins

Alexandre Da Silva Pratas Gabriel

Lucia D'Acunto

Emmanouil Potetsianakis

Robert Koenen

Hendrikus Nathaniël Hindriks

Simon Gunkel

Aschwin Steven Reinier Brandt

Bastiaan Wissingh

Arian Koster

Arjen Veenhuizen

Piotr Wojciech Zuraniewski

Dolf Schinkel

Sylvie Dijkstrasoudarissanane

Sylvie Dijkstra- Soudarissanane

A method of processing video data of a picture is described, the method including: allocating memory for a decoded picture in a decoded picture buffer, DPB, the decoded picture comprising pixels representing video data; receiving a bitstream comprising decoding units, DUs, and storing the DUs in a coded picture buffer, CRB, the DUs representing a coded picture that needs to be decoded into the decoded picture, each of the DUs representing a coded block of pixels; determining if, during decoding of the coded picture, at least one partial output can be performed, the at least one partial output including copying the one or more decoded DUs from the DPB to a data sink, while one or more DUs of the coded picture are not yet decoded and removed the CPB, the one or more decoded DUs representing a part of the decoded picture; and, or performing the at least one partial output if the processor determines that the at least one partial output can be performed, the performing including marking the one or more decoded DUs stored in the DPB as being ready for partial output, the marking signaling the decoder apparatus not to remove the one or more decoded DUs from the DPB; and, copying the one or more marked decoded DUs from the DPB to the data sink without removing the one or more decoded DU from the DPB.

Partial output of a decoded picture buffer in video coding

Methods and apparatus for processing an encoded representation of a sequence of pictures are disclosed, wherein the method includes: allocating memory for a current picture of the sequence of pictures in a decoded picture buffer, DPB; receiving a bitstream comprising an encoded representation of the current picture, and storing the encoded representation as a plurality of decoding units DUs, in a coded picture buffer, CPB, each of the plurality of DUs comprising an encoded set of pixels, wherein no coding dependency exists between coded pixels of different DUs of the current picture; receiving prioritization information associated with each of the plurality of DUs stored in the CPB, the prioritization information defining a decoding order in which the DUs of the current picture are being decoded; decoding one or more of said plurality of DU's into one or more decoded DU's, in accordance with said decoding order, each decoded DU comprising a set of pixels, each set of pixels being associated with a different area in the current picture; storing the sets of pixels in the DPB, a set of pixels being stored at a location in the allocated memory of the DPB that corresponds to a position of the area in the current picture; and, outputting as a partial output, one or more sets of pixels of decoded DUs of the current picture in accordance with the decoding order, while other DUs of the current picture are being decoded or received by the decoding apparatus.

Prioritized decoding and output of parts of a picture in video coding

Methods and apparatus for processing an encoded representation of a sequence of pictures are disclosed, wherein the method includes: a processor of a decoding apparatus allocating memory for one or more pictures in a decoded picture buffer, DPB; the processor receiving a bitstream comprising an encoded representation of a current picture, the encoded representation of the current picture comprising a plurality of decoding units, DUs, each of the plurality of DUs comprising an encoded set of pixels; the processor storing the plurality of DUs of the current picture in a coded picture buffer, CPB; the processor decoding the plurality of DUs into decoded DUs of the current picture and storing the decoded DUs in the allocated memory of the DPB, the processor executing at least one partial output during the decoding of the plurality of DUs of the current picture, the performing of the at least one partial output including: copying one or more of the decoded DUs from the DPB to a data sink before all of the plurality of DUs are decoded, the one or more decoded DUs representing one or more areas of pixels in the current picture; determining output information comprising information about the one or more areas of pixels that are copied to the data sink and information for identifying the current picture; providing access to the output information for use by an application or an application programming interface, API, outside the decoding apparatus.

A method of processing a multi-view video by a client apparatus is described wherein, the multi-view video comprising a set of video streams is created by a set of cameras simultaneously capturing an object in a scene for a set of view angles and for one or more video resolutions, each video stream defining a view associated with a view angle of a camera relative to the object. The method may comprise: receiving a manifest file defining a plurality of windows for the multi-view video, a first window of the plurality of windows defining a first subset of video streams selected from the set of video streams, the first subset of video streams being associated with a first subset of view angles of the set of view angles; determining position information associated with an angular position of a viewer relative to a multi-view display apparatus, the multi-view display apparatus being configured to render the multi-view video identified in the manifest file; selecting the first window from the plurality of windows based on the position information and the first subset of view angles and requesting transmission of the first subset of video streams by a server system; and, receiving multi-view video data associated with the first subset of video streams; and, providing the multi-view video data to a decoder apparatus for decoding.

Multi-view video streaming

One aspect of this disclosure relates a video processing device comprising a processor for processing a manifest file for video streaming for a user. The manifest file comprises at least a plurality of positions defined for a scene that are associated with pre-rendered omnidirectional or volumetric video segments stored on a server system. The manifest file may also contain a plurality of resource locators for retrieving omnidirectional or volumetric video segments from the server system. Each resource locator may be associated with a position defined for the scene. The video processing device may be configured to associate a position of the user with a first position for the scene in the manifest file to retrieve a first omnidirectional or volumetric video segment associated with the first position using a first resource locator from the manifest file.

Video processing device and manifest file for video streaming

A streaming server and a streaming client are described, with the streaming server being configured for streaming video data representing a panoramic video of a scene to the streaming client. The panoramic video may show the scene from a viewpoint within the scene, with the viewpoint being one of a plurality of viewpoints within the scene. When streaming first video data of a first panoramic video, and when starting to stream second video data of a second panoramic video, the streaming client may at least temporarily and simultaneously continue to stream at least part of the first video data by continuing to stream a part of the first video data which represents shared video content of the scene which is visible in the first panoramic video and in the second panoramic video, while additionally streaming second viewpoint-specific video data which represents viewpoint-specific video content of the scene.

Streaming panoramic video of a scene from multiple viewpoints

A method is described for forming output video frames by a decoder device, comprising: receiving a bitstream comprising bitstream parts representing encoded media data of video tiles and tile positioning information associated with the encoded video tiles; extracting tile positioning information from the bitstream, the tile positioning information comprising tile identifiers for identifying encoded video tiles, a tile identifier being associated with one or more boundary identifiers, each identifying a boundary of a video tile identified by a tile identifier; determining a tile map based on the boundary identifiers, the tile map representing a spatial layout of video tiles in an output video frame; and forming an output video frame, including identifying parts of the bitstream representing encoded media data of video tiles identified in the tile map, decoding the identified encoded media data and copying the decoded media data in the output video frame buffer according to spatial layout.

System and methods for video encoding and decoding

Systems and methods for video encoding and decoding

The invention provides a streaming client with a streaming presentation of an event, with said presentation being obtained by dynamically switching between a plurality of media streams which represent different recordings of the event. For that purpose, source data is obtained from a plurality of stream sources. The source data is constituted by content data and/or metadata of concurrent portions of the plurality of media streams. By applying a quality assessment function to the source data, a quality score is assigned to each one of the concurrent portions. This allows selection data to be generated which enables the streaming client to access a selected one of the concurrent portions of a respective media stream. For example, a segment may be selected which provides a highest quality score. By doing so, it is therefore not required anymore for a consumer to manually switch between different media streams.

Establishing a streaming presentation of an event

A processor system and computer-implemented method are provided for generating a composite video stream which may include a non-volumetric video and a volumetric video, at least part of which is to be displayed in front of the non-volumetric video. The volumetric video may be included in the composite video stream in the form of a non-volumetric representation of the volumetric video, for example inserted into a spatial subregion of the non-volumetric video which may be occluded by the volumetric video during display. The encoding, transmission and decoding may thus not have to be modified to support volumetric video. Signaling data may be provided which may be indicative of the composite video stream containing the non-volumetric representation of the volumetric video. A processor system and computer-implemented method may be provided for rendering the composite video stream using the signaling data.

Streaming volumetric and non-volumetric video

A method for adaptive streaming of video is described, wherein the method comprises: transmitting first metadata to a client apparatus, the first metadata defining a first adaptation set, an adaptation set defining one or more sets of segment identifiers, each set of segment identifiers defining a plurality of linearly playable media segments of a predetermined resolution, each media segment in a set of media segments having an identical playback duration, the first metadata enabling a client apparatus to request media segments and sequentially playout the media segments; receiving navigation information from the client apparatus, the navigation information defining at least a first region, in the video frames of the first adaptation set; selecting second metadata defining a second adaptation set on the basis of spatial playback information and a spatial distance or a spatial overlap between the first region and a second region in video frames of the first adaptation set, wherein the spatial playback information signals the server apparatus that the second region is linked to media segments of the second adaptation set; and, transmitting the second metadata or information for retrieving the second metadata to the client apparatus, the second metadata enabling the client apparatus to playback media segments of the second adaptation set after the playback of media segments of the first adaptation set.

Method for adaptive streaming of media

A method of encoding video data into a bitstream comprising: determining from the plurality of video blocks of a video frame of a first resolution a current block and one or more reference blocks including reference samples for predicting original samples of the current block; determining predicted samples of the current block based on a downsampling scheme, the determining comprising computing samples of a first low-resolution block of a second resolution that is lower than the first resolution, and predicting samples of one or more second low-resolution blocks of the second resolution based on the samples of the first low resolution block, the samples of the first low-resolution block and the one or more second low-resolution blocks defining predicted samples of the current Block; and, determining residual samples of a residual block based on original samples of the current block and the predicted samples of the current block; transforming the samples of the residual block into a bitstream; and, embedding metadata associated with the current block in the bitstream for signalling a decoder apparatus that the predicted samples of the current block are based on the downsampling scheme.

Block-level super-resolution based video coding

A method of determining a bitstream is described, the method including: determining a motion vector (MV) of a current block of a current video frame of a sequence of video frames comprising video data captured by a camera, the MV defining an offset of the current block relative to a prediction block, wherein part of the offset is caused by global motion in the video data if the camera is moving relative to a scene; determining motion vector predictor MW candidates for predicting the MV, the MW candidates being determined based on MVs of already encoded blocks; determining a motion compensation vector MCV for MW candidates, determining global motion compensated MW candidates based on a difference between a MW candidate and an associated MCV; selecting a compensated MW; and, determining a motion vector difference MVD based on the compensated MW and the MW; and, using entropy coding to code, the MVD, an indication of the selected compensated MW; and, conversion information for converting a MVP into a compensated MW, into a bitstream.

Video coding based on global motion compensated motion vector predictors

A method of encoding a video frame into a bitstream is described including: applying a downsampling scheme to an original video frame of a first resolution for determining a first low-resolution, LR, original frame and one or more second original LR frames, the first LR original frame and the one or more second LR original frames being of a second resolution that is lower than the first resolution; partitioning the first LR original frame into first original blocks and partitioning the one or more second LR frames into second original blocks; using a prediction method for determining first predicted blocks, the first predicted blocks defining predictions for the first original blocks of the first LR original frame; determining first residual blocks based on the first predicted blocks and the first original blocks and, subsequently, determining first reconstructed blocks based on the first residual blocks and the first predicted blocks; determining second predicted blocks based on the first reconstructed blocks, the second predicted blocks defining predictions for the second original blocks of the one or more second LR original frames; determining second residual blocks based on the second predicted blocks and the second original blocks; and, transforming the first residual blocks and second residual blocks into a bitstream, the bitstream including metadata for signalling a decoder apparatus that during the encoding a downsampling scheme was used for downsampling the video frame.

Frame-level super-resolution-based video coding

A method is described for processing an omnidirectional video by a client device, said omnidirectional video being associated with a 3D coordinate system for determining a spatial position of video data of said omnidirectional video on a curved surface, preferably a sphere, said method comprising: receiving a manifest file, the manifest file comprising a plurality of tile stream identifiers for identifying a plurality of tile streams, the tile streams comprising video frames having image views, whereby the image views of video frames of different tile streams cover different regions of a 2D projection of the omnidirectional video, a region defining a tile; selecting on the basis of spatial relation information in the manifest file and on the basis of a viewpoint of a user of the client device a first tile streams associated with a first resolution and a first tile position and a second tile stream associated with a second resolution and a second tile position, the second resolution being lower than the first resolution, wherein the spatial relation information defines for each tile a tile position and wherein the viewpoint defines a viewing direction of the user, tile position and the viewpoint being defined on the basis of coordinates of the 3D coordinate system.

Growing community of inventors

Delft, Netherlands

Emmanuel Thomas