Apple Inc.

Afshin Taghavi Nasrabadi

Maneli Noorkami

Ranjit Desai

Khaled Mammou

Fabrice A Robinet

Stephen Olivier Lemay

Matan Stauber

Earl M Olson

Wesley M Holder

Michael Stecklein

Negar Nejati

In some embodiments, a computer system displays a caption for a media item at different depths depending on the depth of the portion of the media item over which the caption is displayed. In some embodiments, a computer system displays a user interface element that includes information associated with the media item at different locations relative to the media item depending on attention of the user. In some embodiments, a computer system displays a user interface element that includes information associated with the media item with different visual appearances depending on visual characteristics of the portion of the media item over which the user interface element is displayed.

Methods for displaying user interface elements relative to media content

Various implementations disclosed herein include devices, systems, and methods that generates a stereo view of a three-dimensional (3D) environment by rendering a first eye view and reusing pixels from the first eye view for a second eye view based on a disparity indicative characteristic (e.g., identifying regions of pixel difference/similarity in the views). For example, an example process may include rendering a first view for a first viewpoint (e.g., a left eye view) of a three-dimensional (3D) environment. The process may further include estimating differences between the first view and a second view for a second viewpoint (e.g., a right eye view) of the 3D environment. The process may further include rendering the second view for the second viewpoint based on the estimated differences.

Efficient stereo rendering

A set of planes is determined based on the vertices of a three-dimensional scene. The set of planes is determined independent of any particular orientation with respect to a reference viewpoint of the three-dimensional scene. The vertices are associated with the set of planes and respective textures are associated with the set of planes based on the associated vertices.

Unconstrained multiplane images

A decoding computing device receives a bit stream for compressed volumetric visual content, such as immersive media. The bit stream includes video encoded image frames comprising packed attribute patch images, depth maps for a 3D scene represented by the volumetric visual content, and a variable transparency layer indicating boundaries of objects at different depths within the 3D scene represented by the volumetric visual content. Instead of generating a mesh having a large number of vertices, the decoder generates a lower resolution mesh and applies the variable transparency layer, such that portions of triangles or polygons of the lower resolution mesh extending beyond the boundaries of the objects at the different depths of the 3D scene are made transparent and not shown as being stretched across large depth distances.

Variable transparency layers for light-weight rendering

A decoding computing device determines a focus area of a view of volumetric visual content, such as a three-dimensional (3D) object or scene, and indicates the focus area in a request for compressed volumetric visual content. A server provides a depth map and an attribute atlas that includes attribute and/or texture information for the focus area that is signaled at a higher resolution than other portions of the volumetric visual content. The decoding device also applies one or more mesh simplification techniques, wherein a higher resolution mesh is used for the focus area. The textures/attributes are projected onto the mesh representations, wherein the mesh representations and projected textures are used to reconstruct the volumetric visual content. Hole filling techniques are then applied, wherein a more sophisticated hole filling technique is used in the focus area.

Immersive media content encoding and rendering

Aspects of the subject technology relate to encoding of video frames having content with a variable resolution that varies within the video frame. Aspects of the subject technology can provide an efficient encoding by using smaller macroblocks for lower resolution content within the video frame, and larger macroblocks for higher resolution content within the video frame. An encoder may be provided with resolution information for the content of the video frame, which can be used by the encoder to determine macroblock sizes, macroblock divisions, and/or prediction modes for the encoding of the video frame.

Resolution-based video encoding

A decoding computing device receives a bit stream for compressed 3D volumetric content. The bit stream includes video encoded image frames comprising packed attribute patch images and depth maps for the 3D volumetric content. Instead of generating a mesh having a vertex for each depth value signaled in the depth map, the decoder performs a real-time mesh simplification process to reduce a resolution of the mesh, such that the mesh resolution is reduced without exceeding an error threshold, which may be dynamically determined. Additionally, the decoder may perform a re-meshing of particular regions of the mesh for the 3D volumetric content to avoid cracks or gaps.

Real time simplification of meshes

In one implementation, a camera rig comprises: a first array of image sensors arranged in a planar configuration, wherein the first array of image sensors is provided to capture a first image stream from a first perspective of a physical environment; a second array of image sensors arranged in a non-planar configuration, wherein the second array of image sensors is provided to capture a second image stream from a second perspective of the physical environment different from the first perspective; a buffer provided to store the first and second image streams; and an image processing engine provided to generate a 3D reconstruction of the physical environment based on the first and second image streams.

Method and device for generating a 3D reconstruction of a scene with a hybrid camera rig

An encoder encodes three-dimensional (3D) volumetric content, such as immersive media, using video encoded attribute patch images packed into a 2D atlas to communicate the attribute values for the 3D volumetric content. The encoder also uses mesh-encoded sub-meshes to communicate geometry information for portions of the 3D object or scene corresponding to the attribute patch images packed into the 2D atlas. The encoder applies decimation operations to the sub-meshes to simplify the sub-meshes before mesh encoding the sub-meshes. A distortion analysis is performed to bound the level to which the sub-meshes are simplified at the encoder. Mesh simplification at the encoder reduces the number of vertices and edges included in the sub-meshes which simplifies rendering at a decoder receiving the encoded 3D volumetric content.

3D volumetric content encoding using 2D videos and simplified 3D meshes

One or more computing devices implement a mesh analysis for evaluating meshes to be rendered when rendering immersive content. The mesh analysis identifies objects in a three-dimensional scene and determines geometrical complexity values for the objects. Objects with similar geometrical complexities are grouped into areas and a mesh vertices budget is determined for the respective areas. Metadata indicating the area definitions and corresponding mesh vertices budgets are generated. The metadata may be uploaded to a server to simplify meshes in the scene prior to streaming to a client, or the metadata may be provided to a client for use in simplifying the meshes as part of rendering the scene.

Resolution budgeting by area for immersive video rendering

A system comprises an encoder configured to compress and encode data for three-dimensional volumetric content. The encoder also is configured to segment the three-dimensional volumetric content based on viewing areas, wherein different ones of the viewing areas correspond to visible portions of the volumetric content. The system may provide metadata to a client device to support viewport adaptive rendering of the three-dimensional volumetric content or may adaptively stream portions of the three-dimensional volumetric content to a rending device based on viewing areas of the three-dimensional volumetric content that are to be rendered at the rendering device.

Growing community of inventors

Santa Clara, CA, United States of America

Afshin Taghavi Nasrabadi