Lucasfilm Entertainment Company Ltd.

Michael Koperwas

Frederic P Pighin

Kiran Bhat

Steve Sullivan

Cary Phillips

Nicholas Grant Rasmussen

John Gaeta

Eduardo Hueso

Ji Hun Yu

Sheila Santos

Ronald Mallet

Christopher David Twigg

Rachel Rose

Jeffery Bruce Yost

Yuting Ye

Adrian R Goldenthal

Jung-Seung Hong

Lutz Latta

Julien F Cohen Bengio

Earle M Alexander, Iv

Jeffrey Bruce Yost

Nicholas Rasmussen

Ronald Radeztsky

A method includes causing background content to be displayed on a display device with a first virtual object and a second virtual object; causing augmented reality (AR) content to be rendered based on a location of an AR device relative to the display device; determining that the AR content is in front of the first virtual object in the scene when viewed through the AR device and rendering the background content with a cutout in the first virtual object when the first virtual object overlaps with the AR content; and determining that the AR content is behind the second virtual object in the scene when viewed through the AR device and rendering the AR content with a cutout in the AR content when the AR content overlaps with the second virtual object.

Content presentation and layering across multiple devices

A method may include causing first content to be displayed on a display device; causing second content to be rendered irrespective of a location of a mobile device relative to the display device; and causing the second content to be displayed on the mobile device such that the second content is layered over the first content. When the second content has moved a predetermined distance from the screen, The method may also include causing the second content to be rendered based on the location of the mobile device relative to the display device; and causing the second content to be displayed on the mobile device such that the second content is layered over the first content.

A system includes a computing device that includes a memory configured to store instructions. The system also includes a processor configured to execute the instructions to perform a method that includes receiving multiple representations of one or more expressions of an object. Each of the representations includes position information attained from one or more images of the object. The method also includes producing an animation model from one or more groups of controls that respectively define each of the one or more expressions of the object as provided by the multiple representations. Each control of each group of controls has an adjustable value that defines the geometry of at least one shape of a portion of the respective expression of the object. Producing the animation model includes producing one or more corrective shapes if the animation model is incapable of accurately presenting the one or more expressions of the object as provided by the multiple representations.

Facial animation models

Implementations of the disclosure are directed to generating shadows in the physical world that correspond to virtual objects displayed on MR displays. In some implementations, a method includes: synchronously presenting a version of a scene on each of a MR display system and a projector display system, where during presentation: the MR display system displays a virtual object overlaid over a view of a physical environment; and a projector of the projector display system creates a shadow on a surface in the physical environment, the created shadow corresponding to the virtual object displayed by the MR display. In some implementations, the method includes: loading in a memory of the MR display system, a first version of the scene including the virtual object; and loading in a memory of the projector display system a second version of the scene including a virtual surface onto which the virtual object casts a shadow.

Creating shadows in mixed reality

Some implementations of the disclosure are directed to a pipeline that enables real time engines such as gaming engines to leverage high quality simulations generated offline via film grade simulation systems. In one implementation, a method includes: obtaining simulation data and skeletal mesh data of a character, the simulation data and skeletal mesh data including the character in the same rest pose; importing the skeletal mesh data into a real-time rendering engine; and using at least the simulation data and the imported skeletal mesh data to derive from the simulation data a transformed simulation vertex cache that is usable by the real-time rendering engine during runtime to be skinned in place of the rest pose.

Simulation stream pipeline

A method may include capturing a first image of a physical environment using a mobile device. The mobile device may include a physical camera and a display. The method may also include receiving a second image from a content provider system. The second image may be generated by the content provider system by rendering a view from a virtual camera in a virtual environment. The virtual environment may represent at least a portion of the physical environment. A location of the virtual camera in the virtual environment may correspond to a location of the physical camera in the physical environment. The second image may include a view of a computer-generated object. The method may additionally include generating a third image by compositing the first image and the second image, and causing the third image to be displayed on the display of the mobile device.

Compositing images for augmented reality

A system includes a computing device that includes a memory configured to store instructions. The system also includes a processor configured to execute the instructions to perform a method that includes receiving multiple representations of an object. Each of the representations includes position information of the object and corresponds to an instance in time. For at least one of the representations, the method includes defining a contour that represents a movable silhouette of a surface feature of the object. The method also includes producing a deformable model of the surface of the object from the defined contour and from the at least one representation of the object.

A method may include causing first content to be displayed on a display device. The method may also include determining a location of a mobile device relative to the display device. In some embodiments, the mobile device may be positioned such that the first content is visible to a viewer of the mobile device. The method may additionally include causing second content to be displayed on the mobile device such that the second content is layered over the first content.

A system includes a computing device that includes a memory and a processor configured to execute instructions to perform a method that includes receiving multiple representations of one or more expressions of an object. Each representation includes position information attained from one or more images of the object. The method also includes producing an animation model from one or more groups of controls that respectively define each of the one or more expressions of the object as provided by the multiple representations. Each control of each group of controls has an adjustable value that defines the geometry of at least one shape of a portion of the respective expression of the object. Producing the animation model includes producing one or more corrective shapes if the animation model is incapable of accurately presenting the one or more expressions of the object as provided by the multiple representations.

Performance capture systems and techniques are provided for capturing a performance of a subject and reproducing an animated performance that tracks the subject's performance. For example, systems and techniques are provided for determining control values for controlling an animation model to define features of a computer-generated representation of a subject based on the performance. A method may include obtaining input data corresponding to a pose performed by the subject, the input data including position information defining positions on a face of the subject. The method may further include obtaining an animation model for the subject that includes adjustable controls that control the animation model to define facial features of the computer-generated representation of the face, and matching one or more of the positions on the face with one or more corresponding positions on the animation model. The matching includes using an objective function to project an error onto a control space of the animation model. The method may further include determining, using the projected error and one or more constraints on the adjustable controls, one or more values for one or more of the adjustable controls. The values are configured to control the animation model to cause the computer-generated representation to perform a representation of the pose using the one or more adjustable controls.

Determining control values of an animation model using performance capture

A method may include rendering a first view of a three-dimensional (3-D) virtual scene comprising a view of first content being displayed on a virtual display device from a location in the 3-D virtual scene. The method may also include rendering a second view comprising one or more content objects. The second view may be rendered from the location in the 3-D virtual scene, and the second scene may include a view of the display device as would be seen through a pair of augmented-reality glasses that display the one or more content objects. The method may additionally include generating a composite view by combing the first view and the second view. The method may further include causing the composite view to be displayed on a virtual-reality headset.

Augmented reality simulation

In some embodiments a method of transferring facial expressions from a subject to a computer-generated character is provided where the method includes receiving positional information from a motion capture session of the subject representing a performance having facial expressions to be transferred to the computer-generated character, receiving a first animation model that represents the subject, and receiving a second animation model that represents the computer-generated character. Each of the first and second animation models can include a plurality of adjustable controls that define geometries of the model and that can be adjusted to present different facial expressions on the model, and where the first and second animation models are designed so that setting the same values for the same set of adjustable controls in each model generates a similar facial poses on the models. The method further includes determining a solution, including values for at least some of the plurality of controls, that matches the first animation model to the positional information to reproduce the facial expressions from the performance to the first animation model, retargeting the facial expressions from the performance to the second animation model using the solution; and thereafter, synchronizing lip movement of the second animation model with lip movement from the first animation model.

Lip synchronization between rigs

A system includes a computer system capable of representing one or more animated characters. The computer system includes a blendshape manager that combines multiple blendshapes to produce the animated character. The computer system also includes an expression manager to respectively adjust one or more control parameters associated with each of the plurality of blendshapes for adjusting an expression of the animated character. The computer system also includes a corrective element manager that applies one or more corrective elements to the combined blendshapes based upon at least one of the control parameters. The one or more applied corrective elements are adjustable based upon one or more of the control parameters absent the introduction of one or more additional control parameters.

Growing community of inventors

San Francisco, CA, United States of America

Michael Koperwas