Koninklijke Kpn N.v.

Nederlandse Organisatie Voor Toegepast-Natuurwetenschappelijk Onderzoek Tno

Koninklike Kpn N.v.

Hans Maarten Stokking

Mattijs Oskar Van Deventer

Omar Aziz Niamut

Martin Prins

Fabian Arthur Walraven

Ray Van Brandenburg

Hendrikus Nathaniël Hindriks

Emmanuel Thomas

Victor Bastiaan Klos

John Gerard Beerends

Frank Den Hartog

Alexandre Da Silva Pratas Gabriel

Frank Bart Ter Haar

Simon Gunkel

Simon Norbert Bernhard Gunkel

Peter Joannes Mathias Veugen

Menno Remco Bangma

Anton Havekes

Bernardus Hillen

Robert Koenen

Simon Norbert Bernard Gunkel

Peter-Jan Doets

Frank Theodoor Henk Den Hartog

Eelco Cramer

Robert-Ernst Kooij

Manuel Herrera Van Der Nood

Adrianus Holtzer

Maarten Andreas Hogervorst

Robertus Smets

Hendrikus Nathaniel Hindriks

Gerrit Schilt

Adrianus Cornelis Gerardus Holtzer

Alexandre De Silva Pratas Gabriel

Tessa Esmeé Klunder

Milan Wijnmalen

Nanda Van Der Stap

Karim Mohammed El Assal

Johannes Marinus Maria De Nijs

Yetao Zhu

Anne-Marie Brouwer

Jasper Sebastiaan Van Der Waa

Victor Klos

Frank T H Den Hartog

Frank Theodor Henk Den Hartog

Den Frank Hartog

A method is described for transmitting a multimedia stream from a first terminal and receiving an associated second multimedia stream at a second terminal. The first and second terminal are connected to at least one gateway for enabling the transmission of the stream and the receiving of the associated stream. The method comprises: —initiating the exchange of first multimedia session information between the first terminal and the gateway, using a first protocol; —providing a trigger to the second terminal to initiate the exchange of second multimedia session information between the second terminal and the gateway, using a second protocol; —in response to providing the trigger, the second terminal initiating the exchange of second multimedia session information between the second terminal and the gateway, using a third protocol; —transmitting a first multimedia stream from the first terminal and receiving a second associated multimedia stream at the second terminal.

Method and system for transmitting a multimedia stream

A method of rendering a virtual object by a rendering device includes establishing a communication channel with a mobile device; determining or receiving initial pose information of the mobile device, the pose information defining an initial position and an initial orientation; receiving device information via the communication channel, the device information including orientation, motion, and/or position information for updating the initial pose of the mobile device; using the initial pose information and the device information to determine a current pose of the mobile device, the current pose defining a current position and orientation; receiving first content data from a server system, the first content data representing a virtual object; and, rendering the first content data in the viewport of the rendering device based on the current pose information, the current pose information being used to render the virtual object in spatial alignment with the mobile device.

Rendering a virtual object in spatial alignment with a pose of an electronic device

A combiner system may be provided for combining, in a compressed domain, video streams of different media sources in a composite video stream by including a respective video stream as independently decodable spatial segment(s) in the composite video stream. The combiner system may generate composition metadata describing a composition of the spatial segments in the composite video stream and identification metadata comprising identifiers of the respective video streams. A receiver system may obtain decoded video data of a respective video stream based on the composition metadata and a decoding of the composite video stream, and based on the identification metadata, identify a process for handling the decoded video data. Thereby, the composition of spatial segments may dynamically change, while still allowing the receiver device to correctly handle the spatial segments.

Combining video streams in composite video stream with metadata

A system and computer-implemented method are provided for facilitating a video streaming which comprises a plurality of transmitter devices each transmitting a respective video via a telecommunication network and a receiver device receiving the respective videos via the telecommunication network. The system may comprise an edge node which may receive video from a transmitter device in uncompressed form or in compressed form as a low latency video stream, process the video and encode the processed video as a tile-based video stream. A combiner may then, in the compressed domain, combine any received tile-based video streams to obtain a combined tile-based video stream which contains tiles of the videos of at least two transmitter devices and which may be decodable by a single decoder instance.

Facilitating video streaming and processing by edge computing

A processor system () processes image data from a camera () to render a virtual object in a 3D virtual environment and communicate via a network to a server system (). A processor () obtains image data of the physical space from the camera and second data of a second device (). The processor system sends the image data, the second data and a calibration command to the server system. In the server, a processing resource () processes the data according to the calibration command and sends calibration data indicative of a camera pose and a second pose of the second device. The processor generates 3D object data of the object by processing the image data based on the calibration data. The 3D object data is provided for rendering the virtual object in the 3D virtual environment. So, the processor is in control, while the calibration is executed by the server.

Client/server distributed camera calibration

An orchestration system and method are described for at least in part orchestrating a multidevice video session in which a plurality of devices each transmit a video, for example of a user, via a network and in which a composition system receives said transmitted videos and composites, for a respective device from the plurality of devices, videos of other devices into one or more video streams and transmits the one or more video streams via the network to the device. A grouping of the users/devices may be determined to allow the video stream(s) to be generated for respective devices in a manner which may be dependent on the group to which they are deemed to belong. In particular, the video stream(s) may be generated by the composition system such that the videos of devices within a same group are shown in a better quality than the videos of devices from another group, as a user of a device may prefer to see the users from the same group in a better quality than the users from another group.

Orchestrating a multidevice video session

An encoder system and computer-implemented method may be provided for generating a video stream for a streaming client. The system and method may determine a part of the video which is or would be occluded during display of the video by the streaming client, for example on the basis of signaling data received from the streaming client. A video stream may be generated by, before or as part of encoding of the video, omitting the part of the video, or replacing video data in the part by replacement video data having a lower entropy than said video data. The video stream may be provided to the streaming client, for example via a network. Accordingly, a better compressible version of the video may be obtained, which when displayed by the streaming client, may still contain all or most non-occluded parts visible to a user.

Generating and displaying a video stream by omitting or replacing an occluded part

An intermediary system and method may be provided for assisting a receiver system in processing video data which is streamed as a video stream to the receiver system via a network. The processing of the video data by the receiver system may be dependent on an analysis of the video data. The intermediary system may provide processing assist data to the receiver system which comprises an analysis result or a processing instruction derived from the analysis results. Accordingly, the receiver system may process the video data without a need for the receiver system itself to analyze the video data, thereby offloading computational complexity to the intermediary system. Compared to techniques in which most or all of the processing is performed by the intermediary system, an advantage of continuing to process the video data at the receiver system is that the receiver system may already decode the video stream while the video stream is decoded and/or analyzed by the intermediary system, thereby reducing the delay from transmission to display of the video stream.

Network-based assistance for receiver processing of video data

The invention provides a streaming client with a streaming presentation of an event, with said presentation being obtained by dynamically switching between a plurality of media streams which represent different recordings of the event. For that purpose, source data is obtained from a plurality of stream sources. The source data is constituted by content data and/or metadata of concurrent portions of the plurality of media streams. By applying a quality assessment function to the source data, a quality score is assigned to each one of the concurrent portions. This allows selection data to be generated which enables the streaming client to access a selected one of the concurrent portions of a respective media stream. For example, a segment may be selected which provides a highest quality score. By doing so, it is therefore not required anymore for a consumer to manually switch between different media streams.

Establishing a streaming presentation of an event

A processor system processes image data for rendering a virtual environment for a user present in a real environment. The system receives head tracking data indicative of the orientation of the head of the user. An image processor generates image data for rendering a viewport of the virtual environment on a display system based on the head tracking data. A real-view area is defined in the virtual environment, having at least one boundary. The boundary corresponds to predetermined coordinates in the virtual environment. Thereby a corresponding part of the real environment is made visible in the real-view area, the part showing the real environment as perceived from the user head pose. Effectively the virtual environment is augmented by integrating part of the real environment via the real-view area.

Augmented virtuality self view

A method and processor system are provided which analyze a depth map, which may be obtained from a range sensor capturing depth information of a scene, to identify where an object is located in the scene. Accordingly, a region of interest may be identified in the scene which includes the object, and image data may be selectively obtained of the region of interest, rather than of the entire scene containing the object. This image data may be acquired by an image sensor configured for capturing visible light information of the scene. By only selectively obtaining the image data within the region of interest, rather than all of the image data, improvements may be realized in the computational complexity of a possible further processing of the image data, the storage of the image data and/or the transmission of the image data.

Obtaining image data of an object in a scene

A system and method for rendering a video stream for display on a display device, which may comprise receiving the video stream by streaming and generating display data defining a visible display area to be displayed by the display device, wherein generating of display data comprises rendering the video stream in a sub-area of the visible display area. A selection mechanism may be provided by which a version of the video stream may be selected for streaming having a spatial resolution and/or a bitrate which is selected based on a size of the sub-area.

Rendering video stream in sub-area of visible display area

A processor system and computer-implemented method may be provided for generating a composite video stream which may combine a background video and a foreground video stream into one stream. For that purpose, a spatially segmented encoding of the background video may be obtained, for example in the form of a tiled stream. The foreground video stream may be received, for example, from a(nother) client device. The foreground video stream may be a real-time stream, e.g., when being used in real-time communication. The image data of the foreground video stream may be inserted into the background video by decoding select segments of the background video, inserting the foreground image data into the decoded background image data of these segments, and by encoding the resulting composite image data to obtain composite segments which, together with the non-processed segments of the background video, form a spatially segmented encoding of a composite video.

Generating composite video stream for display in VR

Processor systems and computer-implemented methods are provided for rendering a multiuser virtual environment in which different panoramic images are provided as image-based backdrop for different viewing positions in the virtual environment. When a user switches from a first viewing position for which a first panoramic image is rendered to a second viewing position for which a second panoramic image is rendered, a transition may be rendered for the user, for example as a mathematical transformation of the panoramic images. To avoid other users perceiving the representation of the user in the virtual environment from abruptly switching to the second viewing position, transition data may be provided which enables another processor system to render a representation of the transition in the virtual environment, e.g., by moving an avatar of the user along a movement trajectory.

Generating and signaling transition between panoramic images

A method and processor system are provided for adapting an acoustic rendering of an audio source to a visual rendering of an image-based object. Such visual rendering may be in virtual-reality (VR) or in augmented-reality (AR). The image-based object may be modelled, and an audio object representing the modelled object may be established in an acoustic scene containing the audio source and being associated with the visual rendering. The audio object may be assigned a reverberant and/or absorbent property, and the acoustic rendering may be adapted to the audio object. This way, the acoustic rendering may be adapted to image-based objects, and a user may perceive the acoustic rendering as better matching the visual rendering.

Adapting acoustic rendering to image-based object

Growing community of inventors

Wateringen, Netherlands

Hans Maarten Stokking