Disney Enterprises, Inc.

Pixar

University College London

Eth Zurich (eidgenoessische Technische Hochschule Zurich)

Kenneth John Mitchell

Charalampos Koniaris

Darren Cosker

Malgorzata Edyta Kosek

Robert Walker Sumner

Bochang Moon

Jose A Iglesias-Guitian

Corey D Drake

Nicholas T Swafford

Steven G McDonagh

Martin Klaudiny

David A Sinclair

Iain Matthews

Wojciech Jarosz

Derek Nowrouzezahrai

Rachel E Rodgers

Joseph George Hager, Iv

Kyna P McIntosh

Tim A Weyrich

Ye Pan

Gwyneth Bradbury

Charles Malleson

Ivan Huerta Casado

Yue Li

Derek Edward Bradley

Leonid Sigal

Mark A Reichow

Elliott H Baumbach

Thabo D Beeler

Timothy M Panec

Stelian Coros

Jason A Yeung

Stefan Mangold

Manuel Lang

Giorgio Corbellini

Michael Holton

Sheldon Paul Andrews

John Edgar Park

Rafael C Hostettler

Sheldon Andrews

Eric Smolikowski

Kartic Subr

Dan Andrei Calian

Xiasong Yang

Stefan C Geiger

Thomas Williams

Frederike Dümbgen

Shuang Liu

Llogari Casas Cambra

Floyd M Chitalu

Techniques to facilitate compression of depth data and real-time reconstruction of high-quality light fields. A parameter space of values for a line, pairs of endpoints on different sides of the line, and a palette index for each pixel of a pixel tile of a depth image is sampled. Values for the line, the pairs of endpoints, and the palette index that minimize an error are determined and stored.

Depth codec for real-time, high-quality light field reconstruction

Systems and methods are presented for immersive and simultaneous animation in a mixed reality environment. Techniques disclosed represent a physical object, present at a scene, in a 3D space of a virtual environment associated with the scene. A virtual element is posed relative to the representation of the physical object in the virtual environment. The virtual element is displayed to users from a perspective of each user in the virtual environment. Responsive to an interaction of one user with the virtual element, an edit command is generated and the pose of the virtual element is adjusted in the virtual environment according to the edit command. The display of the virtual element to the users is then updated according to the adjusted pose. When simultaneous and conflicting edit commands are generated by collaborating users, policies to reconcile the conflicting edit commands are disclosed.

Augmenting a physical object with virtual components

Embodiments provide techniques for rendering augmented reality effects on an image of a user's face in real time. The method generally includes receiving an image of a face of a user. A global facial depth map and a luminance map are generated based on the captured image. The captured image is segmented into a plurality of segments. For each segment in the plurality of segments, a displacement energy of the respective segment is minimized using a least square minimization of a linear system for the respective segment. The displacement energy is generally defined by a relationship between a detailed depth map, the global facial depth map and the luminance map. The detailed depth map is generated based on the minimized displacement energy for each segment in the plurality of segments. One or more visual effects are rendered over the captured image using the generated detailed depth map.

Real-time feature preserving rendering of visual effects on an image of a face

Systems, methods, and articles of manufacture are disclosed that enable the compression of depth data and real-time reconstruction of high-quality light fields. In one aspect, spatial compression and decompression of depth images is divided into the following stages: generating a quadtree data structure for each depth image captured by a light field probe and difference mask associated with the depth image, with each node of the quadtree approximating a corresponding portion of the depth image data using an approximating function; generating, from the quadtree for each depth image, a runtime packed form that is more lightweight and has a desired maximum error; and assembling multiple such runtime packed forms into per-probe stream(s); and decoding at runtime the assembled per-probe stream(s). Further, a block compression format is disclosed for approximating depth data by augmenting the block compression format 3DC+(BC4) with a line and two pairs of endpoints.

Embodiments provide for the rendering of illumination effects on real-world objects in augmented reality systems. An example method generally includes overlaying a shader on the augmented reality display. The shader generally corresponds to a three-dimensional geometry of an environment in which the augmented reality display is operating, and the shader generally comprises a plurality of vertices forming a plurality of polygons. A computer-generated lighting source is introduced into the augmented reality display. One or more polygons of the shader are illuminated based on the computer-generated lighting source, thereby illuminating one or more real-world objects in the environment with direct lighting from the computer-generated lighting source and reflected and refracted lighting from surfaces in the environment.

Introducing real-time lighting effects to illuminate real-world physical objects in see-through augmented reality displays

Embodiments provide for animation streaming for media interaction by receiving, at a generator, inputs from a target device presenting of a virtual environment; updating, based on the user inputs, a model of the virtual environment; determining network conditions between the generator and target device; generating a packet that includes a forecasted animation set for a virtual object in the updated model that comprises rig updates for the virtual object for at least two different states, and a number of states included in the packet is based on the network conditions; and streaming the packet to the target device, where the target device: receives a second input to interact with the virtual environment that changes the virtual environment to a given state; selects and applies a rig update associated with the given state a local model of the virtual object; and outputs the updated local model on the target device.

Animation streaming for media interaction

Techniques for selectively removing Monte Carlo (MC) noise from a geometric buffer (G-buffer). Embodiments identify the G-buffer for rendering an image of a three-dimensional scene from a viewpoint. Embodiments determine, for each of a plurality of pixels in the image being rendered, respective world position information based on the three-dimensional scene and a position and orientation of the viewpoint. A pre-filtering operation is then performed to selectively remove the MC noise from the G-buffer, based on the determined world position information for the plurality of pixels.

Noise reduction on G-buffers for Monte Carlo filtering

Training data from multiple types of sensors and captured in previous capture sessions can be fused within a physics-based tracking framework to train motion priors using different deep learning techniques, such as convolutional neural networks (CNN) and Recurrent Temporal Restricted Boltzmann Machines (RTRBMs). In embodiments employing one or more CNNs, two streams of filters can be used. In those embodiments, one stream of the filters can be used to learn the temporal information and the other stream of the filters can be used to learn spatial information. In embodiments employing one or more RTRBMs, all visible nodes of the RTRBMs can be clamped with values obtained from the training data or data synthesized from the training data. In cases where sensor data is unavailable, the input nodes may be unclamped and the one or more RTRBMs can generate the missing sensor data.

Deep-learning motion priors for full-body performance capture in real-time

Presented herein are systems and methods configured to generate virtual entities representing real-world users. In some implementations, the systems and/or methods are configured to capture user appearance information with imaging devices and sensors, determines correspondence values conveying correspondences between the appearance of the user's body or user's head and individual ones of default body models and/or default head models, modifies a set of values defining a base body model and/or base head model based on determined correspondence values and sets of base values defining the default body models and/or default head models. The base body model and/or base head model may be modified to model the appearance of the body and/or head of the user.

Single shot capture to animated VR avatar

Techniques for constructing a three-dimensional model of facial geometry are disclosed. A first three-dimensional model of an object is generated, based on a plurality of captured images of the object. A projected three-dimensional model of the object is determined, based on a plurality of identified blendshapes relating to the object. A second three-dimensional model of the object is generated, based on the first three-dimensional model of the object and the projected three dimensional model of the object.

Dense reconstruction for narrow baseline motion observations

Systems, methods, and articles of manufacture for real-time rendering using compressed animated light fields are disclosed. One embodiment provides a pipeline, from offline rendering of an animated scene from sparse optimized viewpoints to real-time rendering of the scene with freedom of movement, that includes three stages: offline preparation and rendering, stream compression, and real-time decompression and reconstruction. During offline rendering, optimal placements for cameras in the scene are determined, and color and depth images are rendered using such cameras. Color and depth data is then compressed using an integrated spatial and temporal scheme permitting high performance on graphics processing units for virtual reality applications. The compressed content may be decoded and reconstructed in real-time by selecting, using heuristics, cameras that provide useful data for a viewer, selecting grid cells from those cameras that are visible to the viewer, and using a ray marching technique to reconstruct the scene.

Real-time rendering with compressed animated light fields

A system for performing memory allocation for seamless media content presentation includes a computing platform having a CPU, a GPU having a GPU memory, and a main memory storing a memory allocation software code. The CPU executes the memory allocation software code to transfer a first dataset of media content to the GPU memory, seamlessly present the media content to a system user, register a location of the system user during the seamless presentation of the media content, and register a timecode status of the media content at the location. The CPU further executes the memory allocation software code to identify a second dataset of the media content based on the location and the timecode status, transfer a first differential dataset to the GPU memory, continue to seamlessly present the media content to the system user, and transfer a second differential dataset out of the GPU memory.

Memory allocation for seamless media content presentation

Single shot capture to animated vr avatar

Growing community of inventors

Earlston, United Kingdom

Kenneth John Mitchell