Apple Inc.

Meta Platforms Technologies, LLC

Pol Pla I Conesa

Aaron Mackay Burns

Alexis Henri Palangie

William A Sorrentino, Iii

Michal Hlavac

Stephen Olivier Lemay

Paul Xiaopeng Wang

Alan C Dye

Bas Ording

Jeffrey M Faulkner

Jonathan R Dascola

Wan Si Wan

Richard R Dellinger

Giancarlo Yerkes

Pedro Mari

Israel Pastrana Vicente

David M Schattel

Nathan Gitter

Aaron P Thompson

Peter D Anton

Shih-Sang Chiu

Christopher D Mckenzie

Ryan P Brooks

Philipp Rockel

Gregory M Apodaca

Jesse Chand

Benjamin Hunter Boesel

Dorian D Dargan

Jay Moon

Kristi E Bauerly

Peter L Hajas

Yeliz Karadayi

Robert T Tilton

Aaron S Faucher

Nathan Aschenbach

Wesley M Holder

Earl M Olson

Jonathan Ravasz

Wai Leong Chak

Roger Ibars Martinez

Chris Rojas

Bryan Sparks

Daniel Rosas

Zoey C Taylor

Thomas Reardon

Evgenii Krivoruchko

Emron Jackson Henry

Miquel Estany Rodriguez

James J Owen

Kristi E S Bauerly

Nicholas W Henderson

Ian Roth

Jenny Kam

Jie Gu

John Jacob Blakeley

Michael Ishigaki

Kristi Es Bauerly

Jossie E Tirado Arroyo

Joseph N Bravate

Eric Ma

Vikram Tank

Jonathan Michael Proto

Paul Mealy

Kevin Harper

Johnathon Simmons

William A Iii Sorrentino

Shi-Sang Chiu

A non-transitory computer-readable storage medium including instructions for distributing tasks between a handheld intermediary processing device (HIPD) and communicatively coupled devices is disclosed. The instructions cause one or more processors to, while a HIPD and a head-wearable device are communicatively coupled and share operational data for performing one or more computational tasks, identify one or more back-end tasks and one or more front-end tasks associated with performing the one or more computational tasks. The instructions further cause performance of the one or more back-end tasks at the HIPD including updating the operational data to create updated operational data. The instructions further cause performance of the one or more front-end tasks at the head-wearable device using the updated operational data such that a representation based on the one or more computational tasks is presented to the user by the head-wearable device.

Intermediary device for offloading processing operations for an artificial-reality system and for enabling interactions and input paradigms, and systems and methods of use thereof

A computer system detects a hand at a first position in a first configuration while displaying a view of a three-dimensional environment. In response, the computer system displays a first visual indication at a first display location in the three-dimensional environment. The computer system detects a movement of the hand while displaying the first visual indication. In response, the computer system displays the first indication at a second display location if the movement maintains the configuration of the hand and moves the hand as a whole to a second position; or the computer system ceases to display the first visual indication and displays a second visual indication if the movement changes the configuration of the hand to a second configuration at the first position.

Devices, methods, and graphical user interfaces for interacting with three-dimensional environments

Aspects of the present disclosure are directed to triggering virtual keyboard selections using multiple input modalities. An interface manager can display an interface, such as a virtual keyboard, to a user in an artificial reality environment. Implementations of the interface manager can track user eye gaze input and user hand input (e.g., hand or finger motion). The interface manager can resolve a character selection on the virtual keyboard according to the tracked user gaze input based on detection that the user's hand motion meets a trigger criteria. For example, the interface manager can: detect that the tracked user hand motion meets the trigger criteria at a given point in time; and resolve a selection from the virtual keyboard (e.g., selection of a displayed character) according to the tracked user gaze on the virtual keyboard at the given point in time.

Virtual selections using multiple input modalities

In accordance with some embodiments, an exemplary process for controlling representations of virtual objects based on respective user contexts that each correspond to different respective locations in a computer-generated reality (CGR) environment is described.

Controlling representations of virtual objects in a computer-generated reality environment

Aspects of the disclosure are directed to an interface for receiving input using multiple modalities in an artificial reality environment. The interface can be a virtual keyboard displayed in an artificial reality environment that includes characters arranged as elements. Implementations include an artificial reality device/system for displaying the artificial reality environment and receiving user input in a first modality, and a controller device for receiving user input in an additional input modality. For example, the artificial reality system can be configured to receive user gaze input as a first input modality and the controller device can be configured to receive input in a second modality, such as touch input received at a trackpad. An interface manager can process input in the multiple modalities to control an indicator on the virtual interface. The interface manager can also resolve character selections from the virtual interface according to the input.

Artificial reality input using multiple modalities

Exemplary processes are described, including processes to move and/or resize user interface elements in a computer-generated reality environment.

Controlling displays

A computer system concurrently displays a view of a physical environment; and a computer-generated user interface element overlaid on the view of the physical environment. An appearance of the computer-generated user interface element is based on an appearance of the view of the physical environment on which the computer-generated user interface element is overlaid. In response to an appearance of the physical environment changing, the appearance of the computer-generated user interface element is updated at a first time based on a graphical composition of the appearance of one or more portions of the physical environment at different times prior to the first time, including: an appearance of a first portion of the physical environment at a second time that is before the first time; and an appearance of a second portion of the physical environment at a third time that is before the second time.

While displaying a three-dimensional scene including at least a first virtual object displayed with a first value corresponding to a first portion and a second value corresponding to a second portion of the first virtual object at a first location and a first physical surface at a second location, a computer system generates a first visual effect at the second location of the three-dimensional scene, including modifying a visual appearance of a first portion of the first physical surface in the three-dimensional scene in accordance with the first value for the first display property that corresponds to the first portion of the first virtual object; and modifying a visual appearance of a second portion of the first physical surface in the three-dimensional scene in accordance with the second value for the first display property that corresponds to the second portion of the first virtual object.

In some embodiments, an electronic device enhances interactions with virtual objects in a three-dimensional environment. In some embodiments, an electronic device enhances interactions with selectable user interface elements. In some embodiments, an electronic device enhances interactions with slider user interface elements. In some embodiments, an electronic device moves virtual objects in a three-dimensional environment and facilitates accessing actions associated with virtual objects.

Methods for interacting with virtual controls and/or an affordance for moving virtual objects in virtual environments

In some embodiments, an electronic device facilitates cursor interactions in different regions in a three-dimensional environment. In some embodiments, an electronic device facilitates cursor interactions in content. In some embodiments, an electronic device facilitates cursor movement. In some embodiments, an electronic device facilitates interaction with multiple input devices. In some embodiments, a computer system facilitates cursor movement based on movement of a hand of a user of the computer system and a location of a gaze of the user in the three-dimensional environment. In some embodiments, a computer system facilitates cursor selection and scrolling of content in the three-dimensional environment.

Methods for cursor-based interactions with an electronic device

Virtual keyboard selections using multiple input modalities

A virtual object system can orchestrate virtual objects defined as a collection of components and with inheritance in an object hierarchy. Virtual object components can include a container, data, a template, and a controller. A container can define the volume the virtual object is authorized to write into. A virtual object's data can specify features such as visual elements, parameters, links to external data, meta-data, etc. The template can define view states of the virtual object and contextual breakpoints for transitioning between them. Each view state can control when and how the virtual object presents data elements. The controller can define logic for the virtual object to respond to input, context, etc. The definition of each object can specify which other object in an object hierarchy that object extends, where extending an object includes inheriting that object's components, which can be modified or overwritten as part of the extension.

Virtual object structures and interrelationships

The present disclosure provides world-controlled augments and application-controlled augments. World-controlled augments can be controlled directly by a shell in the artificial reality environment. To allow even inexperienced users to develop world-controlled augments, a world-controlled builder system is provided. Application-controlled augments may be resource intensive (e.g., using eye-tracking, social-media tie-ins, etc.), may support complicated interactions among themselves, may require or have extensive use of inputs and permissioned resources, and are controlled by their hosting application. When a running application is halted, the application closes its application-controlled augments but can choose to have the XR system run a 'place-holder' world-controlled augment for as long as the application is not running. The place-holder world-controlled augment preserves the appearance of the application in the artificial reality environment but uses few system resources. Upon a user selecting a place-holder world-controlled augment, the XR system can run the associated application and close the place-holder world-controlled augment.

World-controlled and application-controlled augments in an artificial-reality environment

The present disclosure relates to techniques for displaying representations of physical input devices and overlaying visual features on the representations of physical input devices in a computer-generated reality (CGR) environment. The techniques include displaying a virtual application in a CGR environment and, in response to detecting an input field in the displayed virtual application, displaying at least a portion of the displayed application on a representation of a physical input device. The at least a portion of the displayed application includes the detected input field. In response to detecting an input received at the physical input device, the input field is updated with the input, and the updated input field is displayed.

Displaying physical input devices as virtual objects

Growing community of inventors

Belmont, CA, United States of America

Pol Pla I Conesa