Apple Inc.

Other

Olivier Soares

Ian M Richter

Amritpal Singh Saini

Michael J Rockwell

Andrew P Mason

Yang Song

Andrew R Harvey

Graham L Fyffe

Gilles M Cadet

Shaobo Guan

Martin E Johnson

Antti Vanne

Theo Auclair

Vignesh Ganapathi Subramanian

Fletcher R Rothkopf

Grant H Mulliken

Izzet B Yildiz

Lilli I Jonsson

Akiko Ikkai

Haarm-Pieter Duiker

John S McCarten

Ran Luo

Thomas Gebauer

Generating a 3D representation of a subject includes obtaining a set of images of a subject. For each sample point, a classifier value is obtained based on each image. The classifier value indicates a relationship of the sample point to an interior or exterior of a volume of the subject. In addition, deformation data is determined for the subject across the image. The classifier values are fused based on the deformation data, and a 3D occupation field is determined for the subject based on the fused classifier values.

Dynamic PIFu enrollment

An audio system and a method of determining an audio filter based on a position of an audio device of the audio system, are described. The audio system receives an image of the audio device being worn by a user and determines, based on the image and a known geometric relationship between a datum on the audio device and an electroacoustic transducer of the audio device, a relative position between the electroacoustic transducer and an anatomical feature of the user. The audio filter is determined based on the relative position. The audio filter can be applied to an audio input signal to render spatialized sound to the user through the electroacoustic transducer, or the audio filter can be applied to a microphone input signal to capture speech of the user by the electroacoustic transducer. Other aspects are also described and claimed.

Audio system and method of determining audio filter based on device position

In one implementation, a method includes presenting, via a display device, a first synthesized reality (SR) view of an event that includes SR content associated with the event. The SR content includes a plurality of related layers of SR content that perform actions associated with the event. The method includes detecting, via one or more input devices, selection of a respective layer among the plurality of related layers of SR content associated with the event. The method includes presenting, via the display device, a second SR view of the event that includes the respective layer of SR content in response to the selection of the respective layer. The second SR view corresponds to a point-of-view of the respective layer.

Method and devices for presenting and manipulating conditionally dependent synthesized reality content threads

Various implementations disclosed herein include devices, systems, and methods for presenting objective-effectuators in synthesized reality settings. In various implementations, a device includes a non-transitory memory and one or more processors coupled with the non-transitory memory. In some implementations, a method includes instantiating an objective-effectuator into a synthesized reality setting. In some implementations, the objective-effectuator is characterized by a set of predefined actions and a set of visual rendering attributes. In some implementations, the method includes obtaining an objective for the objective-effectuator. In some implementations, the method includes determining contextual information characterizing the synthesized reality setting. In some implementations, the method includes generating a sequence of actions from the set of predefined actions based on the contextual information and the objective. In some implementations, the method includes modifying the objective-effectuator based on the sequence of actions.

Objective-effectuators in synthesized reality settings

Estimating emotion may include obtaining an image of at least part of a face, and applying, to the image, an expression convolutional neural network ('CNN') to obtain a latent vector for the image, where the expression CNN is trained from a plurality of pairs each comprising a facial image and a 3D mesh representation corresponding to the facial image. Estimating emotion may further include comparing the latent vector for the image to a plurality of previously processed latent vectors associated with known emotion types to estimate an emotion type for the image.

Emotion detection

Rendering an avatar may include determining an expression to be represented by an avatar, obtaining a blood texture map associated with the expression, wherein the blood texture map represents an offset of coloration from an albedo map for the expression, and rendering the avatar utilizing the blood texture map.

Machine learning-based blood flow tracking

Various implementations disclosed herein include devices, systems, and methods that present a view of a device user's face portion, that would otherwise be blocked by an electronic device positioned in front of the face, on an outward-facing display of the user's device. The view of the user's face portion may be configured to enable observers to see the user's eyes and facial expressions as if they were seeing through a clear device at the user's actual eyes and facial expressions. Various techniques are used to provide views of the user's face that are realistic, that show the user's current facial appearance, and/or that present the face portion with 3D spatial accuracy, e.g., each eye appearing to be in its actual 3D position. Some implementations combine live data with previously-obtained data, e.g., combining live data with enrollment data.

Rendering of enrolled user's face for external display

Various implementations disclosed herein include devices, systems, and methods that present a view of a device user's face portion, that would otherwise be blocked by an electronic device positioned in front of the face, on an outward-facing display of the user's device. The view of the user's face portion may be configured to enable observers to see the user's eyes and facial expressions as if they were seeing through a clear device at the user's actual eyes and facial expressions. Various techniques are used to provide views of the user's face that are realistic, that show the user's current facial appearance, and/or that present the face portion with 3D spatial accuracy, e.g., each eye appearing to be in its actual 3D position.

Rendering of a guest user's face for external display

Sensor data indicating a user's response to an avatar experience in which the user experiences a rendered avatar model is obtained. A perceptual quality metric value corresponding to the rendered avatar model is determined based on the sensor data and a determined relationship between the sensor data and the perceptual quality metric value. The avatar model is re-rendered for display based on the perceptual quality metric value.

Perceptually and physiologically constrained optimization of avatar models

In one implementation, a method includes: instantiating a first objective-effectuator (OE) associated with first attributes and a second OE associated with second attributes into a synthesized reality (SR) setting, wherein the first OE is encapsulated within the second OE; providing a first objective to the first OE based on the first and second attributes; providing a second objective to the second OE based on the second attributes, wherein the first and second objectives are associated with a time period between a first and second temporal points; generating a first set of actions for the first OE based on the first objective and a second set of actions for the second OE based on the second objective; and rendering for display the SR setting for the time period including the first set of actions performed by the first OE and the second set of actions performed by the second OE.

Rendering an avatar in a selected environment may include determining as inputs into an inferred shading network, an expression geometry to be represented by an avatar, head pose, and camera angle, along with a lighting representation for the selected environment. The inferred shading network may then generate a texture of a face to be utilized in rendering the avatar. The lighting representation may be obtained as lighting latent variables which are obtained from an environment autoencoder trained on environment images with various lighting conditions.

Growing community of inventors

Oakland, CA, United States of America

Olivier Soares