Nederlandse Organisatie Voor Toegepast-natuurwetenschappelijk Onderzoek Tno

Koninklijke Kpn N.v.

Koninklike Kpn N.v.

Nederlandse Ogranisatie Voor Toegepast-natuurwetenschappelijk Onderzoek Tno

Hendrikus Nathaniël Hindriks

Hans Maarten Stokking

Emmanuel Thomas

Martin Prins

Mattijs Oskar Van Deventer

Alexandre Da Silva Pratas Gabriel

Frank Bart Ter Haar

Robert Koenen

Simon Gunkel

Emmanouil Potetsianakis

Simon Norbert Bernhard Gunkel

Johannes Marinus Maria De Nijs

Karim Mohammed El Assal

Alexandre De Silva Pratas Gabriel

A method of processing video data of a picture is described, the method including: allocating memory for a decoded picture in a decoded picture buffer, DPB, the decoded picture comprising pixels representing video data; receiving a bitstream comprising decoding units, DUs, and storing the DUs in a coded picture buffer, CRB, the DUs representing a coded picture that needs to be decoded into the decoded picture, each of the DUs representing a coded block of pixels; determining if, during decoding of the coded picture, at least one partial output can be performed, the at least one partial output including copying the one or more decoded DUs from the DPB to a data sink, while one or more DUs of the coded picture are not yet decoded and removed the CPB, the one or more decoded DUs representing a part of the decoded picture; and, or performing the at least one partial output if the processor determines that the at least one partial output can be performed, the performing including marking the one or more decoded DUs stored in the DPB as being ready for partial output, the marking signaling the decoder apparatus not to remove the one or more decoded DUs from the DPB; and, copying the one or more marked decoded DUs from the DPB to the data sink without removing the one or more decoded DU from the DPB.

Partial output of a decoded picture buffer in video coding

A system and computer-implemented method are provided for facilitating a video streaming which comprises a plurality of transmitter devices each transmitting a respective video via a telecommunication network and a receiver device receiving the respective videos via the telecommunication network. The system may comprise an edge node which may receive video from a transmitter device in uncompressed form or in compressed form as a low latency video stream, process the video and encode the processed video as a tile-based video stream. A combiner may then, in the compressed domain, combine any received tile-based video streams to obtain a combined tile-based video stream which contains tiles of the videos of at least two transmitter devices and which may be decodable by a single decoder instance.

Facilitating video streaming and processing by edge computing

An encoder system and computer-implemented method may be provided for generating a video stream for a streaming client. The system and method may determine a part of the video which is or would be occluded during display of the video by the streaming client, for example on the basis of signaling data received from the streaming client. A video stream may be generated by, before or as part of encoding of the video, omitting the part of the video, or replacing video data in the part by replacement video data having a lower entropy than said video data. The video stream may be provided to the streaming client, for example via a network. Accordingly, a better compressible version of the video may be obtained, which when displayed by the streaming client, may still contain all or most non-occluded parts visible to a user.

Generating and displaying a video stream by omitting or replacing an occluded part

An intermediary system and method may be provided for assisting a receiver system in processing video data which is streamed as a video stream to the receiver system via a network. The processing of the video data by the receiver system may be dependent on an analysis of the video data. The intermediary system may provide processing assist data to the receiver system which comprises an analysis result or a processing instruction derived from the analysis results. Accordingly, the receiver system may process the video data without a need for the receiver system itself to analyze the video data, thereby offloading computational complexity to the intermediary system. Compared to techniques in which most or all of the processing is performed by the intermediary system, an advantage of continuing to process the video data at the receiver system is that the receiver system may already decode the video stream while the video stream is decoded and/or analyzed by the intermediary system, thereby reducing the delay from transmission to display of the video stream.

Network-based assistance for receiver processing of video data

Processor systems and computer-implemented methods are provided for rendering a multiuser virtual environment in which different panoramic images are provided as image-based backdrop for different viewing positions in the virtual environment. When a user switches from a first viewing position for which a first panoramic image is rendered to a second viewing position for which a second panoramic image is rendered, a transition may be rendered for the user, for example as a mathematical transformation of the panoramic images. To avoid other users perceiving the representation of the user in the virtual environment from abruptly switching to the second viewing position, transition data may be provided which enables another processor system to render a representation of the transition in the virtual environment, e.g., by moving an avatar of the user along a movement trajectory.

Generating and signaling transition between panoramic images

A method and processor system are provided for adapting an acoustic rendering of an audio source to a visual rendering of an image-based object. Such visual rendering may be in virtual-reality (VR) or in augmented-reality (AR). The image-based object may be modelled, and an audio object representing the modelled object may be established in an acoustic scene containing the audio source and being associated with the visual rendering. The audio object may be assigned a reverberant and/or absorbent property, and the acoustic rendering may be adapted to the audio object. This way, the acoustic rendering may be adapted to image-based objects, and a user may perceive the acoustic rendering as better matching the visual rendering.

Adapting acoustic rendering to image-based object

Systems and methods for screen sharing are provided. The screen sharing may comprise providing image data from a sender system to a receiver system. The receiver system may be a Virtual Reality [VR] system configured to render the image data in a virtual environment by projecting the image data onto a virtual canvas. At the sender system, visual content may be identified to share with the receiver system. The rendering or display of the virtual environment by the receiver may impose legibility constraints on the visual content. An adjusted version of the visual content may be generated which provides improved legibility when viewed in the virtual environment. The image data representing the adjusted version may then be provided to the receiver system. Accordingly, the legibility of the visual content shared by the sender system may be improved when rendered, displayed and subsequently viewed by a user in the virtual environment.

Screen sharing for display in VR

Systems and methods are provided for generating and rendering an enhanced audiovisual recording of a user, which may be used for multiuser communication, e.g., in Virtual Reality. Such an enhanced recording may be generated by determining a face orientation of the user in the audiovisual recording, and generating orientation data specifying an orientation which represents said determined face orientation. During rendering, the audio data may be rendered based on the orientation data, namely by rendering the audio data as a spatial audio source having a spatial direction which is congruent with the face orientation of the user in a visual representation of the user. Accordingly, the spatial direction of the voice of the user may better match the user's face direction in the user's visual representation.

Growing community of inventors

The Hague, Netherlands

Hendrikus Nathaniël Hindriks