Apple Inc.

Other

Cydesign, Inc.

Mark E Drummond

Siva Chandra Mouli Sivapurapu

Bo Morgan

Payal Jotwani

Edward S Ahn

Aashi Manglik

Behrooz Mahasseni

Dan Feng

Daniel Laszlo Kovacs

Mu Qiao

Shaun D Budhram

Michael J Gutensohn

Peter Meier

Ian M Richter

Bradley Warren Peebler

Cameron J Dunn

John Christopher Russell

Daniel Ulbricht

Serdar Uckun

Angela Blechschmidt

Michelle Chua

Adam Michael O'Hern

Shem Nguyen

Novaira Masood

Noah Jonathan Gamboa

Noah Gamboa

William Benjamin Schaefer, Iv

Mark Flanagan Ward

Kerry Ryan Poppa

Michael Theodore Koopmans

Lee Travis Bergmann Johnson

James Moll

A device obtains a request to display a graphical environment. The graphical environment is associated with a set of saliency values corresponding to respective portions of the graphical environment. A POV for displaying the graphical environment is selected based on the set of saliency values. The graphical environment is displayed from the selected POV on the display.

Displaying an environment from a selected point-of-view

In one implementation, a method of composing a scene content is performed at a device including a display, one or more processors, and non-transitory memory. The method includes generating a definition of a scene based on textual or speech input and a model of a physical environment, wherein the definition includes a constraint that defines a spatial relationship between a virtual asset and an anchor asset that corresponds to one or more physical objects in the physical environment. The method includes generating, based on the definition of the scene and the model of the physical environment, a first instance of the scene that satisfies the constraint with the virtual asset in the spatial relationship with a first one of the one or more physical objects in the physical environment. The method includes presenting, on the display, the first instance of the scene.

Methods and systems for composing and executing a scene

In some implementations, a method includes obtaining an objective for a computer-generated reality (CGR) representation of an objective-effectuator. In some implementations, the objective is associated with a plurality of time frames. In some implementations, the method includes determining a plurality of candidate plans that satisfy the objective. In some implementations, the method includes selecting a first candidate plan of the plurality of candidate plans based on a selection criterion. In some implementations, the method includes effectuating the first candidate plan in order to satisfy the objective. In some implementations, the first candidate plan triggers the CGR representation of the objective-effectuator to perform a series of actions over the plurality of time frames associated with the objective.

Planner for an objective-effectuator

Various implementations disclosed herein include devices, systems, and methods for granular motion control for a virtual agent. In various implementations, a device includes a non-transitory memory and one or more processors coupled with the non-transitory memory. In some implementations, a method includes obtaining an action for a virtual agent. In some implementations, the action is associated with a plurality of time frames. In some implementations, the method includes, for a first time frame of the plurality of time frames, determining respective confidence scores for a plurality of granular motions that advance the virtual agent towards completion of the action. In some implementations, the method includes selecting a subset of the plurality of granular motions based on the respective confidence scores.

Granular motion control for a virtual agent

In some implementations, a device includes one or more processors and a non-transitory memory. In some implementations, a method includes obtaining an object that includes a plurality of portions including a first portion and a second portion. In some implementations, the method includes determining a first saliency value for the first portion of the object and a second saliency value for the second portion of the object. In some implementations, the method includes triggering presentation of the first portion of the object in accordance with a first display characteristic that is based on the first saliency value and the second portion of the object in accordance with a second display characteristic that is based on the second saliency value.

Triggering presentation of an object based on saliency

Various implementations disclosed herein include a method performed by an electronic device including one or more processors and a non-transitory memory. The method includes obtaining first environmental data that characterizes a first physical setting that is spatially distinct from a second physical setting. The second physical setting includes the electronic device. The method includes generating, using the first environmental data, a first extended reality (XR) environment corresponding to the first physical setting. The method includes generating a first mapping between a first bounding surface of the second physical setting and a first bounding surface of the first XR environment.

Connecting spatially distinct settings

In some implementations, a method includes obtaining, by a virtual intelligent agent (VIA), a perceptual property vector (PPV) for a graphical representation of a physical element. In some implementations, the PPV includes one or more perceptual characteristic values characterizing the graphical representation of the physical element. In some implementations, the method includes instantiating a graphical representation of the VIA in a graphical environment that includes the graphical representation of the physical element and an affordance that is associated with the graphical representation of the physical element. In some implementations, the method includes generating, by the VIA, an action for the graphical representation of the VIA based on the PPV. In some implementations, the method includes displaying a manipulation of the affordance by the graphical representation of the VIA in order to effectuate the action generated by the VIA.

Responding to representations of physical elements

According to various implementations, a method is performed at an electronic device including one or more processors, a non-transitory memory, and a display. The method includes displaying, on the display, a virtual agent that is associated with a first viewing frustum. The first viewing frustum includes a user avatar associated with a user, and the user avatar includes a visual representation of one or more eyes. The method includes, while displaying the virtual agent associated with the first viewing frustum, obtaining eye tracking data that is indicative of eye behavior associated with an eye of the user, updating the visual representation of one or more eyes based on the eye behavior, and directing the virtual agent to perform an action based on the updating and scene information associated with the electronic device.

Directing a virtual agent based on eye behavior of a user

In one implementation, a method for recording an XR environment. The method includes: presenting, via the display device, a graphical environment with one or more virtual agents, wherein the graphical environment corresponds to a composition of extended reality (XR) content, including the one or more virtual agents, and an image stream of a physical environment captured from a first point-of-view (POV) of the physical environment; detecting, via the one or more input devices, a user input selecting a first virtual agent from among the one or more virtual agents; and in response to detecting the user input, recording a plurality of data streams associated with the graphical environment including a first image stream of the graphical environment from the first POV and one or more data streams of the graphical environment from a current POV of the first virtual agent.

Method and device for debugging program execution and content playback

In one implementation, a method of displaying content is performed at a device including a display, one or more processors, and non-transitory memory. The method includes scanning a first physical environment to detect a first physical object in the first physical environment and a second physical object in the first physical environment, wherein the first physical object meets at least one first object criterion and the second physical object meets at least one second object criterion. The method includes displaying, in association with the first physical environment, a virtual object moving along a first path from the first physical object to the second physical object. The method includes scanning a second physical environment to detect a third physical object in the second physical environment and a fourth physical object in the second physical environment, wherein the third physical object meets the at least one first object criterion and the fourth physical object meets the at least one second object criterion. The method includes displaying, in association with the second physical environment, the virtual object moving along a second path from the third physical object to the fourth physical object, wherein the second path is different than the first path.

A method includes displaying, via a display device, a virtual agent moving according to a motion type. The virtual agent is defined by a plurality of virtual joints and motions of the virtual agent are controllable by providing a corresponding plurality of torques to the plurality of virtual joints. The method includes, while the virtual agent is moving according to the motion type, registering an interaction event with the virtual agent. The interaction event initiates a change to the motion type. The method includes, in response to registering the interaction event, generating, using a motion controller, a plurality of torque values for the virtual agent based on a function of the motion type and the interaction event, and generating an animation for the virtual agent by providing the plurality of torque values to the plurality of virtual joints of the virtual agent.

Controlling joints using learned torques

Various implementations disclosed herein include devices, systems, and methods for training of an action determining component of a computer character. In some implementations, actions are taken by the character in a 3D environment according to an action determining component of the character, where the character is rewarded or penalized for interactions associated with an object/concept in the 3D environment according to an assigned object/concept reward or penalty. In some implementations, the reward or the penalty assigned to the object/concept is modified, and the character is then rewarded or penalized for interactions associated with the object/concept according to the modified reward or the modified penalty. The action determining component of the character is trained using a reinforcement learning technique that accounts for rewards or penalties obtained by virtual character for interactions associated with the object/concept.

Training a character through interactions

In one implementation, a method of achieving an objective using a scene graph includes identifying a set of transition couples of a plurality of transition couples between sets of spatial relationships of the scene graph, wherein each spatial relationship of a respective first set of spatial relationships indicated by each transition couple of the set of transition couples is included in an initial set of spatial relationships or an respective second set of spatial relationships indicated by a previous transition couple and wherein a particular spatial relationship is included in a respective second set of spatial relationships of a last transition couple of the set of transition couples.

Scene graph assisted navigation

In some implementations, a method includes obtaining a semantic construction of a physical environment. In some implementations, the semantic construction of the physical environment includes a representation of a physical element and a semantic label for the physical element. In some implementations, the method includes obtaining a graphical representation of the physical element. In some implementations, the method includes synthesizing a perceptual property vector (PPV) for the graphical representation of the physical element based on the semantic label for the physical element. In some implementations, the PPV includes one or more perceptual characteristic values characterizing the graphical representation of the physical element. In some implementations, the method includes compositing an affordance in association with the graphical representation of the physical element. In some implementations, the affordance allows interaction with the graphical representation of the physical element in accordance with the perceptual characteristic values included in the PPV.

Perceptual property vector for an object

In one implementation, a method of defining a negative space in a three-dimensional scene model is performed at a device including a processor and non-transitory memory. The method includes obtaining a three-dimensional scene model of a physical environment including a plurality of points, wherein each of the plurality of points is associated with a set of coordinates in a three-dimensional space. The method includes defining a subspace in the three-dimensional space with less than a threshold number of the plurality of points. The method includes determining a semantic label for the subspace. The method includes generating a characterization vector of the subspace, wherein the characterization vector includes the spatial extent of the subspace and the semantic label.

Growing community of inventors

Palo Alto, CA, United States of America

Mark E Drummond