Dolby Laboratories Licensing Corporation

Netlogic Microsystems, Inc

Meta Platforms Technologies, LLC

Luminous Networks, Inc.

Ajit Ninan

Wenhui Jia

Chun Chi Wan

Gregory John Ward

Neil Mammen

Arkady Ten

Chaitanya Atluru

Haricharan Lakshman

Scott J Daly

Tyrome Y Brown

William Rozzi

Robin Atkins

Michael E Miller

Timo Kunkel

Alexandre Chapiro

Shwetha Ram

Samir N Hulyalkar

Ashley Penna

Shane Mario Ruggieri

James Thomas Triplett

Domagoj Baricevic

Gaven Wang

Qifan Huang

Jasper Chao

Helge Seetzen

Jason C Fan

Michael E Ichiriu

Sanjay Agrawal

Martin Fabry

Larry A Wall

Ankur Shah

Matthew Chang

Gregory Buschek

Huiming Tong

Qiaoli Yang

Deon Poncini

Ten Arkady

Alexander Y Levitskiy

Vijay Kamarshi

Saliency regions are identified in a global scene depicted by volumetric video. Saliency video streams that track the saliency regions are generated. Each saliency video stream tracks a respective saliency region. A saliency stream based representation of the volumetric video is generated to include the saliency video streams. The saliency stream based representation of the volumetric video is transmitted to a video streaming client.

Representing volumetric video in saliency video streams

Images of a user's head are acquired at a plurality of different orientational angles through image sensors operating in conjunction with a media consumption system. The acquired images of the user's head are used to select or predict a specific personalized head related transfer function for the user. Spatial audio rendered by audio speakers operating in conjunction with the media consumption system is adjusted or modified based at least in part on the specific personalized HRTF selected for the user.

Head tracking and HRTF prediction

Images of an actual rendering environment are acquired through image sensors operating in conjunction with a media consumption system. The acquired images of the actual rendering environment are used to predict audio characteristics of objects present in the actual rendering environment. Spatial audio rendered, to a user in the actual rendering environment, by audio speakers operating in conjunction with the media consumption system is adjusted or modified based at least in part on the audio characteristics of the objects present in the actual rendering environment.

Device and rendering environment tracking

An eyewear device comprises a left lens assembly and a right lens assembly. The left lens assembly includes a left focus tunable lens and a left focus fixed lens. A right lens assembly includes a right focus tunable lens and a right focus fixed lens. The eyewear device may be used in 3D display applications, virtual reality applications, augmented reality applications, remote presence applications, etc. The eyewear device may also be used as vision correction glasses.

Eyewear devices with focus tunable lenses

At a first time point, a first light capturing device at a first spatial location in a three-dimensional (3D) space captures first light rays from light sources located at designated spatial locations on a viewer device in the 3D space. At the first time point, a second light capturing device at a second spatial location in the 3D space captures second light rays from the light sources located at the designated spatial locations on the viewer device in the 3D space. Based on the first light rays captured by the first light capturing device and the second light rays captured by the second light capturing device, at least one of a spatial position and a spatial direction, at the first time point, of the viewer device is determined.

Passive multi-wearable-devices tracking

A non-random-access video stream is received. A first image block is encoded after second image blocks according to a non-random-access processing order. View direction data is received to indicate a viewer's view direction coinciding with a location covered by the first image block. The first image block is encoded into the random-access video stream before the second image blocks in a random-access processing order. The random-access video stream is delivered to a recipient decoding device operated by the viewer to cause the first image block to be processed and rendered before the second image blocks according to the random-access processing order.

Supporting view direction based random access of bitstream

An eye tracking technique obtains a more precise estimate of a gaze location on an image by determining the vergence location of a user's eyes. The more precise estimate of the gaze location may be obtained by using multiple inputs, including a coarse estimate of the gaze location using a camera that receives non-visible light reflected from the eyes, discrete probabilities of gaze locations for each eye, depth information for objects contained in the image, and saliency information for objects contained in the image.

Eye tracking based on vergence

Video images are rendered in viewports of content viewers. Each content viewer views video images through a respective viewport in the viewports. Spatial locations in the video images to which foveal visions of the content viewers are directed are determined. ROIs in the video images are identified based on the spatial locations in the video images.

Predicting and verifying regions of interest selections

First foviated images are streamed to a streaming client. The first foviated images with first image metadata sets are used to generate first display mapped images for rendering to a viewer at first time points. View direction data is collected and used to determine a second view direction of the viewer at a second time point. A second foviated image and a second image metadata set are generated from a second HDR source image in reference to the second view direction of the viewer and used to generate a second display mapped image for rendering to the viewer at the second time point. The second image metadata set comprises a display management metadata portions for adapting a focal-vision and peripheral-vision image portions to corresponding image portions in the second display mapped image. The focal-vision display management metadata portion is generated with a predicted light adaptation level of the viewer for the second time point. The second foviated image and the second image metadata set are transmitted to the video streaming client.

Foviation and HDR

Spatial information that describes spatial locations of visual objects as in a three-dimensional (3D) image space as represented in one or more multi-view unlayered images is accessed. Based on the spatial information, a cinema image layer and one or more device image layers are generated from the one or more multi-view unlayered images. A multi-layer multi-view video signal comprising the cinema image layer and the device image layers is sent to downstream devices for rendering.

Layered augmented entertainment experiences

A spatial direction of a wearable device that represents an actual viewing direction of the wearable device is determined. The spatial direction of the wearable device is used to select, from a multi-view image comprising single-view images, a set of single-view images. A display image is caused to be rendered on a device display of the wearable device. The display image represents a single-view image as viewed from the actual viewing direction of the wearable device. The display image is constructed based on the spatial direction of the wearable device and the set of single-view images.

Adapting video images for wearable devices

A device and method for video rendering. The device includes a memory and an electronic processor. The electronic processor is configured to receive, from a source device, video data including multiple reference viewpoints, determine a target image plane corresponding to a target viewpoint, determine, within the target image plane, one or more target image regions, and determine, for each target image region, a proxy image region larger than the corresponding target image region. The electronic processor is configured to determine, for each target image region, a plurality of reference pixels that fit within the corresponding proxy image region, project, for each target image region, the plurality of reference pixels that fit within the corresponding proxy image region to the target image region, producing a rendered target region from each target image region, and composite one or more of the rendered target regions to create video rendering.

Multi-resolution multi-view video rendering

A wearable device for augmented media content experiences can be formed with a mountable physical structure that has removably mountable positions and component devices that are removably mounted through the removably mountable positions. The component devices can be specifically selected based on a specific type of content consumption environment in which the wearable device is to operate. The mountable physical structure may be subject to a device washing process to which the component devices are not subject to, after the wearable device including the mountable physical structure and the component devices is used by a viewer in a content consumption session in the specific type of content consumption environment, so long as the component devices are subsequently removed from the mountable physical structure.

Modular and detachable wearable devices for AR/VR/MR

Based on viewing tracking data, a viewer's view direction to a three-dimensional (3D) scene depicted by a first video image is determined. The first video image has been streamed in a video stream to the streaming client device before the first time point and rendered with the streaming client device to the viewer at the first time point. Based on the viewer's view direction, a target view portion is identified in a second video image to be streamed in the video stream to the streaming client device to be rendered at a second time point subsequent to the first time point. The target view portion is encoded into the video stream with a higher target spatiotemporal resolution than that used to encode remaining non-target view portions in the second video image.

Positional zero latency

While a viewer is viewing a first stereoscopic image comprising a first left image and a first right image, a left vergence angle of a left eye of a viewer and a right vergence angle of a right eye of the viewer are determined. A virtual object depth is determined based at least in part on (i) the left vergence angle of the left eye of the viewer and (ii) the right vergence angle of the right eye of the viewer. A second stereoscopic image comprising a second left image and a second right image for the viewer is rendered on one or more image displays. The second stereoscopic image is subsequent to the first stereoscopic image. The second stereoscopic image is projected from the one or more image displays to a virtual object plane at the virtual object depth.

Growing community of inventors

San Jose, CA, United States of America

Ajit Ninan