Disney Enterprises, Inc.

Mitsubishi Electric Research Laboratories, Inc.

Nvidia Corporation

Eth Zurich (eidgenoessische Technische Hochschule Zurich)

Eth Zurich (eidgenossische Technische Hochschule Zurich)

Hewlett-packard Development Company, L.p.

Eth Zurich

Dybuster Ag

Other

Technion Research & Development Foundation Limited

Pixar

Eidgenossische Technische Hochschule Zurich

Eth Zuerich

Swiss Federal Institute of Technology Zurich

Liberovision Ag

Markus H Gross

Alexander Sorkine Hornung

Wojciech Matusik

Hanspeter Pfister

Robert Walker Sumner

Thabo D Beeler

Bernd Bickel

Derek Edward Bradley

Manuel Lang

Bernhard Thomaszewski

Matthias Zwicker

Matthias Müller-Fischer

Changil Kim

Bruno Heidelberger

Matthias Teschner

Umeshwar Dayal

Ming C Hao

Jeroen Van Baar

Simon Heinzle

Philipp Kraehenbuehl

Johannes Schmid

Thomas Holenstein

Meichun Hsu

Oliver Wang

Philip John Jackson

Stelian Coros

Paul A Beardsley

Aljosa A Smolic

Henning Zimmer

Kenny Mitchell

Mubbasir Kapadia

Thomas Oskam

Yael Pritch

Sasha Anna Schriber

Simon Schirm

Miguel A Otaduy

Stephen Robert Marschner

Remo Ziegler

Huw David Bowles

Peter Kaufmann

Stephan Würmlin

Merada Richter

Isa Simo

Nils Thüerey

Pascal Bérard

Tiberiu Popa

Christian Voegeli

Edouard Lamboray

Tim Weyrich

Mark Arana

Alan Hersh Karp

Christopher Richard Schroers

Anthony M Accardo

Mark Joseph Meyer

Marc Stewart Levoy

Gioacchino Noris

Thomas R Gross

Abdelaziz Djelouah

Wojciech Jarosz

Nikolce Stefanoski

Ilya Baran

Prashanth Chandran

Stefan Mangold

Brian McWilliams

Aljosa Aleksej Andrej Smolic

Peter Charles Barnum

Alexander Sorkine-Hornung

Maryann Simmons

Rasmus Tamstorf

Ahmet Cengiz Öztireli

Roberto Aiello

Chenglei Wu

Moritz Niklaus Bächer

Matthias Heinz Muller-Fischer

Craig Gotsman

Steven Poulakos

Nicola V Ranieri

Jean-Charles Bazin

Richard P Hammersley

Yeara Kozlov

Bochang Moon

Kenneth Mitchell

Simone Meyer

Wai Kit Addy Ngan

Szymon Marek Rusinkiewicz

Fabian Hahn

Forrester Cole

Marcel Germann

Richard Keiser

Rolf Adelsberger

Daniel Cotting

Christian Regg

Andreas G P Hubeli

Vittorio Megaro

Chris Thomes

Huw Bowles

Ulrich Mueller

Martin Sebastian Senn

Yongzhe Wang

Michael Waschbüsch

Systems and methods are provided for a workflow framework that scriptwriters can utilize when developing (live-action/animation/cinematic) virtual reality (VR) experiences or content. A script can be parsed to identify one or more elements in a script, and a VR representation of the one or more elements can be automatically generated. A user may develop or edit the script which can be presented in a visual and temporal manner along with the VR representation. The user may edit the VR representation, and the visual and temporal presentation of the script can be commensurately represented. The script may be analyzed for consistency and/or cohesiveness in the context of the VR representation or experience. A preview of the VR experience or content can be generated from the script and/or the VR representation.

Virtual reality experience scriptwriting

According to one implementation, a pose estimation and body tracking system includes a computing platform having a hardware processor and a system memory storing a software code including a tracking module trained to track motions. The software code receives a series of images of motion by a subject, and for each image, uses the tracking module to determine locations corresponding respectively to two-dimensional (2D) skeletal landmarks of the subject based on constraints imposed by features of a hierarchical skeleton model intersecting at each 2D skeletal landmark. The software code further uses the tracking module to infer joint angles of the subject based on the locations and determine a three-dimensional (3D) pose of the subject based on the locations and the joint angles, resulting in a series of 3D poses. The software code outputs a tracking image corresponding to the motion by the subject based on the series of 3D poses.

Pose estimation and body tracking using an artificial neural network

A system includes a memory configured to store a plurality of tools and a processor configured to use the plurality of tools. The processor uses the plurality of tools to retrieve historical information relating to a storyline, retrieve user information relating to user interactions with the storyline, provide the historical information and the user information for incorporating into the content, and generate the content using the historical information and the user information.

Systems and methods for non-linear content creation

According to one implementation, a video processing system includes a computing platform having a hardware processor and a system memory storing a frame interpolation software code, the frame interpolation software code including a convolutional neural network (CNN) trained using a loss function having an image loss term summed with a phase loss term. The hardware processor executes the frame interpolation software code to receive first and second consecutive video frames including respective first and second images, and to decompose the first and second images to produce respective first and second image decompositions. The hardware processor further executes the frame interpolation software code to use the CNN to determine an intermediate image decomposition corresponding to an interpolated video frame for insertion between the first and second video frames based on the first and second image decompositions, and to synthesize the interpolated video frame based on the intermediate image decomposition.

Video frame interpolation using a convolutional neural network

A system and method for non-invasive reconstruction of an entire object-specific or person-specific teeth row from just a set of photographs of the mouth region of an object (e.g., an animal) or a person (e.g., an actor or a patient) are provided. A teeth statistic model defining individual teeth in a teeth row can be developed. The teeth statistical model can jointly describe shape and pose variations per tooth, and as well as placement of the individual teeth in the teeth row. In some embodiments, the teeth statistic model can be trained using teeth information from 3D scan data of different sample subjects. The 3D scan data can be used to establish a database of teeth of various shapes and poses. Geometry information regarding the individual teeth can be extracted from the 3D scan data. The teeth statistic model can be trained using the geometry information regarding the individual teeth.

Model-based teeth reconstruction

A computer-implemented method is provided for physical face cloning to generate a synthetic skin. Rather than attempt to reproduce the mechanical properties of biological tissue, an output-oriented approach is utilized that models the synthetic skin as an elastic material with isotropic and homogeneous properties (e.g., silicone rubber). The method includes capturing a plurality of expressive poses from a human subject and generating a computational model based on one or more material parameters of a material. In one embodiment, the computational model is a compressible neo-Hookean material model configured to simulate deformation behavior of the synthetic skin. The method further includes optimizing a shape geometry of the synthetic skin based on the computational model and the captured expressive poses. An optimization process is provided that varies the thickness of the synthetic skin based on a minimization of an elastic energy with respect to rest state positions of the synthetic skin.

Physical face cloning

Methods, systems, and computer-readable memory are provided for determining time-varying anatomical and physiological tissue characteristics of an animation rig. For example, shape and material properties are defined for a plurality of sample configurations of the animation rig. The shape and material properties are associated with the plurality of sample configurations. An animation of the animation rig is obtained, and one or more configurations of the animation rig are determined for one or more frames of the animation. The determined one or more configurations include shape and material properties, and are determined using one or more sample configurations of the animation rig. A simulation of the animation rig is performed using the determined one or more configurations. Performing the simulation includes computing physical effects for addition to the animation of the animation rig.

Methods and systems of enriching blendshape rigs with physical simulation

Systems and techniques for generating a parametric eye model of one or more eyes are provided. The systems and techniques may include obtaining eye data from an eye model database. The eye data includes eyeball data and iris data corresponding to a plurality of eyes. The systems and techniques may further include generating an eyeball model using the eyeball data. Generating the eyeball model includes establishing correspondences among the plurality of eyes. The systems and techniques may further include generating an iris model using the iris data. Generating the iris model includes sampling one or more patches of one or more of the plurality of eyes using an iris control map and merging the one or more patches into a synthesized texture. The systems and techniques may further include generating the parametric eye model that includes the eyeball model and the iris model.

Methods and systems of generating a parametric eye model

Systems and techniques for reconstructing one or more eyes using a parametric eye model are provided. The systems and techniques may include obtaining one or more input images that include at least one eye. The systems and techniques may further include obtaining a parametric eye model including an eyeball model and an iris model. The systems and techniques may further include determining parameters of the parametric eye model from the one or more input images. The parameters can be determined to fit the parametric eye model to the at least one eye in the one or more input images. The parameters include a control map used by the iris model to synthesize an iris of the at least one eye. The systems and techniques may further include reconstructing the at least one eye using the parametric eye model with the determined parameters.

Methods and systems of performing eye reconstruction using a parametric model

Embodiments can provide adaptive image filtering. Under this approach, image quality can be enhanced by adjusting an approximation function to better adapt image signals in different parts of an image. Certain parts of the image may be enhanced using a certain approximation function while some other parts of the image may be enhanced using a different approximation function. In certain embodiments, the approximation function selected for a part of the image can be a polynomial function having a specific order. The specific polynomial order can be applied directly to obtain an estimated image value of the part of the image. In certain embodiments, the estimation of the reconstruction error can include iteratively estimating a bias term of the reconstruction error and a variance term of the reconstruction error.

Adaptive polynomial rendering

Methods and systems for generating stereoscopic content with granular control over binocular disparity based on multi-perspective imaging from representations of light fields are provided. The stereoscopic content is computed as piecewise continuous cuts through a representation of a light field, minimizing an energy reflecting prescribed parameters such as depth budget, maximum binocular disparity gradient, desired stereoscopic baseline. The methods and systems may be used for efficient and flexible stereoscopic post-processing, such as reducing excessive binocular disparity while preserving perceived depth or retargeting of already captured scenes to various view settings. Moreover, such methods and systems are highly useful for content creation in the context of multi-view autostereoscopic displays and provide a novel conceptual approach to stereoscopic image processing and post-production.

Multi-perspective stereoscopy from light fields

The disclosure provides an approach for estimating depth in a scene. According to one aspect, regions where the depth estimation is expected to perform well may first be identified in full-resolution epipolar-plane images (EPIs) generated from a plurality of images of the scene. Depth estimates for EPI-pixels with high edge confidence are determined by testing a number of discrete depth hypotheses and picking depths that lead to highest color density of sampled EPI-pixels. The depth estimate may also be propagated throughout the EPIs. This process of depth estimation and propagation may be iterated until all EPI-pixels with high edge confidence have been processed, and all EPIs may also be processed in this manner. The EPIs are then iteratively downsampled to coarser resolutions, at which edge confidence for EPI-pixels not yet processed are determined, depth estimates of EPI-pixels with high edge confidence made, and depth estimates propagated throughout the EPIs.

Scene reconstruction from high spatio-angular resolution light fields

A method for image processing in video conferencing, for correcting the gaze of an interlocutor in an image or a sequence of images captured by at least one real camera, comprises the steps of

Method and system for image processing in video conferencing

In an object generation system, consumable base materials are characterized in a characterization process wherein an object generation system can use a plurality of so-characterized base materials. User input representing a desired object and set of characteristics for that desired object are processed, using a computer or computing device, to derive a mapping of locations for placement of portions of the plurality of base materials such that when the mapping is provided to an object generator, the generated object approximates the representing a desired object and set of characteristics. The characterization of a base material might include elasticity of the base material, the user input might be a desired shape and elasticity, the object generator might be a 3D multi-material printer and the generated object might at least approximate the desired shape and elasticity as a result of being constructed from the plurality of base materials used by the printer.

Fabrication of materials with desired characteristics from base materials having determined characteristics

Growing community of inventors

Uster, Switzerland

Markus H Gross