Nokia Technologies Oy

Varjo Technologies Oy

Nokia Corporation

Nokia Technology Oy

Kimmo Tapio Roimela

Vinod Kumar Malamal Vadakital

Mikko Strandborg

Yu You

Lixin Fan

Lauri Ilola

Sebastian Schwarz

Lukasz Kondrad

Arto Juhani Lehtiniemi

Mika Pesonen

Johannes Pystynen

Antti Johannes Eronen

Juha Henrik Arrasvuori

Jussi Severi Uusitalo

Junsheng Fu

Miikka Tapani Vilermo

Lauri Aleksi Ilola

Ranjeeth Ravindra Shetty

Johannes Rajala

Sujeet Shyamsundar Mate

Jussi Artturi Leppänen

Emre Baris Aksu

Jussi Leppanen

Ville-Veikko Mattila

Petros Belimpasakis

Ville Timonen

Jussi Leppänen

Jaakko Keränen

Tomi Aarnio

Jaakko Olli Taavetti Keränen

Joonas Itäranta

Lauri Iiola

Jaakko Keranen

Pekka Väänänen

Jaakko KERÄNEN

Disclosed is method including: obtaining neural network(s) trained for rendering images, wherein input of neural network(s) has 3D position of point in real-world environment and output of neural network(s) includes colour and opacity of point; obtaining 3D model(s) of real-world environment; receiving viewpoint from perspective of which image is to be generated; receiving gaze direction; determining region of real-world environment that is to be represented in image, based on viewpoint and field of view of image; determining gaze portion and peripheral portion of region of real-world environment, based on gaze direction, wherein gaze portion corresponds to gaze direction, while peripheral portion surrounds gaze portion; utilising neural network(s) to ray march for gaze portion, to generate gaze segment of image; and utilising 3D model(s) to generate peripheral segment of image.

Foveated rendering using neural radiance fields

A hierarchical data structure has sets of nodes representing a 3D space of an environment at different granularity levels. Sets of neural networks at different granularity levels are trained. For a portion of an output image, a granularity level at which the portion is to be reconstructed is determined. A corresponding node is identified; the node having sets of child nodes. A set of child nodes is selected at the granularity level at which the portion is to be reconstructed. For a child node, a cascade of neural networks is utilised to reconstruct the portion. Granularity level of N+1neural network is higher than that of Nneural network. Input of a neural network includes outputs of at least a predefined number of previous neural networks.

Multiresolution neural networks for 3D reconstruction

A system and method for receiving colour images, depth images and viewpoint information; dividing 3D space occupied by real-world environment into 3D grid(s) of voxels; create 3D data structure(s) comprising nodes, each node representing corresponding voxel; dividing colour image and depth image into colour tiles and depth tiles, respectively; mapping colour tile to voxel(s) whose colour information is captured in colour tile; storing, in node representing voxel(s), viewpoint information indicative of viewpoint from which colour and depth images are captured, along with any of: colour tile that captures colour information of voxel(s) and corresponding depth tile that captured depth information, or reference information indicative of unique identification of colour tile and corresponding depth tile; and utilising 3D data structure(s) for training neural network(s), wherein input of neural network(s) comprises 3D position of point and output of neural network(s) comprises colour and opacity of point.

Multiresolution truncated neural radiance fields

A method and technical equipment for encoding, where the method comprises at least receiving a video presentation frame, where the video presentation represents a three-dimensional data in the form of mesh data (); separating from the mesh data information on vertices, wherein the information comprises at least connectivity data defining connections between vertices (); determining parameters relating to said connectivity data (); encoding the parameters to a first bitstream as a video component (); and storing the encoded first bitstream for transmission to a rendering apparatus (). In addition to encoding, also decoding is disclosed.

Method, an apparatus and a computer program product for volumetric video encoding and decoding

Disclosed is a method and system for: obtaining 3D data structure comprising nodes, each node representing voxel of 3D grid of voxels, wherein node stores viewpoint information, with any of: (i) colour tile that captures colour information of voxel and depth tile, (ii) reference information indicative of unique identification of colour and depth tiles; utilising 3D data structure for training neural network(s), wherein input of neural network(s) comprises 3D position of point in real-world environment and output of neural network(s) comprises colour and opacity of point; and for new viewpoint, determining visible nodes whose voxels are visible from new viewpoint; for visible node, selecting depth tile(s) whose viewpoint(s) matches new viewpoint most closely; reconstructing 2D geometry of objects from depth tiles; and utilising neural network(s) to render colours for pixels of output colour image.

Image-based environment reconstruction with view-dependent colour

A method including: receiving colour images, depth images, and viewpoint information; dividing 3D space occupied by real-world environment into 3D grid(s) of voxels (); creating 3D data structure(s) comprising nodes, each node representing corresponding voxel; dividing colour image and depth image into colour tiles and depth tiles, respectively; mapping colour tile to voxel(s) whose colour information is captured in colour tile, based on depth information captured in corresponding depth tile and viewpoint from which colour image and depth image are captured; and storing, in node representing voxel(s), reference information indicative of unique identification of colour tile that captures colour information of voxel(s) and corresponding depth tile that captures depth information, along with viewpoint information indicative of viewpoint from which colour image and depth image are captured.

Image-tiles-based environment reconstruction

The embodiments relate to a method comprising receiving as an input a volumetric video frame comprising volumetric video data (); decomposing the volumetric video frame into one or more patches, wherein a patch comprises a volumetric video data component (); packing several patches, where at least two patches of the several patches comprise a different volumetric video data component with respect to each other, into one video frame (); generating a bitstream comprising an encoded video frame (); signaling, in or along the bitstream, existence of encoded video frame containing patches of more than one different volumetric video data component (); and transmitting the encoded bitstream to a storage for rendering (). The embodiments also relate to a technical equipment for implementing the method.

Method, an apparatus and a computer program product for volumetric video encoding and video decoding

A method comprising: providing a 3D representation of at least one object as an input for an encoder (); projecting the 3D representation onto at least one 2D patch (); generating at least a geometry image and a texture image from the 2D patch (); generating, based on the geometry image, a mesh comprising a number of vertices (); mapping the number of vertices to two-dimensional (2D) coordinates of the texture image (), and signalling said 2D coordinates of the texture image to be applied to the number of vertices of the mesh in or along a bitstream ().

Apparatus, a method and a computer program for volumetric video

A method for decoding comprises receiving a compressed bitstream relating to a video presentation; decoding from the received bitstream data relating to three-dimensional blocks of a video frame; for each block of the video frame, determining information on surface parameters; generating a bounding box for a three-dimensional block according to the surface parameters; calculating a ray direction for each pixel of the three-dimensional block from a viewpoint to coordinates of the pixel; determining at least two points according to an intersection of the ray to the generated bounding box; ray casting points between the determined at least two points until a condition to finish the ray casting is met; reconstructing three-dimensional data from the geometry image and the texture image according to the information on one or more surface parameters for a block; reconstructing a video presentation according to the reconstructed three-dimensional data for rendering.

A computer-implemented method including: receiving visible-light images captured from viewpoints using visible-light camera(s); creating 3D model of real-world environment, wherein 3D model stores colour information pertaining to 3D points on surfaces of real objects (); dividing 3D points into groups of 3D points, based on at least one of: whether surface normal of 3D points in group lie within predefined threshold angle from each other, differences in materials of real objects, differences in textures of surfaces of real objects; for group of 3D points, determining at least two of visible-light images in which group of 3D points is captured from different viewpoints, wherein said images are representative of different surface irradiances of group of 3D points; and storing, in 3D model, information indicative of different surface irradiances.

Image reconstruction with view-dependent surface irradiance

An example embodiment relates to a method comprising: receiving (), by a client device, information about a scene associated with a 3D presentation for initiating viewport rendering by an application, wherein said scene is hosted by a server as one or more encoded 2D patches, said 2D patches being projections of the 3D representation of the scene; providing () the server with information for selecting 2D patches required to reconstruct at least one segment of the 3D scene as visible in a given viewport; obtaining (), from the server, one or more video bitstreams generated based on the selected 2D patches and related metadata for extracting said 2D patches from said one or more video bitstreams; and rendering (), by the client device, the given viewport as a sub-segment of the 3D representation of the scene based on said one or more 2D patches and the related metadata.

A method, apparatus, and computer program product provide for compression of volumetric video in a manner that is more efficient and requires less metadata updates. A method can include converting a volumetric video scene into a canonical representation comprising a plurality of frames comprising a video atlas and corresponding metadata. Each frame can be divided into a plurality of tiles based on different tile characteristics, which can be used to analyze and characterize the different tiles for atlas packing purposes. Tiles can be clipped or merged or otherwise edited in the atlas. Tiles can be decomposed to metadata suitable for later rendering of the tile. Tiles having a similar enough color attribute or depth attribute during the entire group of photos can be rendered as a single instance, clipped entirely, and rendered based upon a reference tile and the stored metadata.

Signalling of metadata for volumetric video

Apparatuses, methods, and computer programs are disclosed to implement video-based cloud compression model to world signaling. An example apparatus includes at least one processor; and at least one non-transitory memory including computer program code; wherein the at least one memory and the computer program code are configured to, with the at least one processor, cause the apparatus at least to perform: provide first signaling information comprising information related to a world domain, wherein the world domain is a point cloud frame that is represented by a number of points in a first volumetric coordinate system; and provide second signaling information comprising information related to a conversion of a model domain to the world domain, wherein the model domain represents the point cloud frame by a number of points in a second volumetric coordinate system.

Video-based point cloud compression model to world signaling information

The various embodiments of the invention relates to a method and a technical equipment for video compression. The method comprises processing volumetric image data comprising a plurality of points; defining a hemisphere () around a surface () normal at each point in the volumetric image; partitioning each of the defined hemispheres spatially into a predefined number of angles (); determining a representative radiance value for each angle of the predefined number of angles () of a defined hemisphere (); generating a matrix () for a point storing the determined representative radiance values; and encoding the generated matrix () for the point for video compression.

Method and technical equipment for encoding and decoding volumetric video

A method comprising: processing a recording of a scene to recognise a predetermined user command event performed within the scene; and automatically controlling image processing of a captured image of the scene to adapt the captured image, in dependence on said recognition of the predetermined user command event performed within the scene.

Growing community of inventors

Tampere, Finland

Kimmo Tapio Roimela