Meta Platforms Technologies, LLC

Facebook Technologies, LLC

Tanya Renee Jonker

Aakar Gupta

Hrvoje Benko

Daniel John Wigdor

Ting Zhang

Stephanie Santosa

Marcello Giordano

Tovi Samuel Grossman

Sean Edwin Trowbridge

Michael Glueck

Rorik Henrikson

Carolina Brum Medeiros

Daniel Clarke

Benjamin LaFreniere

Ruta Parimal Desai

Kashyap Todi

Tianyi Wang

Joao Marcelo Evangelista Belo

Maozheng Zhao

Ran Tan

Alec Pierce

Sebastian Freitag

Naveen Sendhilnathan

Difeng Yu

Felix Izarra

Jacqueline Fashimpaur

Peiqi Tang

Disclosed herein is an extended reality system, and associated techniques, whereby personalized usage data of a user in one or more extended reality environments over a period of time can be collected and provided to a predictive model to determine optimal shortcut assignments for presentation to and subsequent use by the user. Determining the optimal shortcut assignments can involve estimating a plurality of interaction times for the user in the one or more extended reality environments, generating an optimized graphical user interface within the one or more extended reality environments with the optimal shortcut assignments reflected in the optimized graphical user interface, and rendering the optimized graphical user interface in the one or more extended reality environments to the user. The system may collect additional personalized usage data and periodically update the optimal shortcut assignments based on ongoing use of the system by the user.

Designing and optimizing adaptive shortcuts for extended reality

The disclosed computer-implemented method may include systems and methods for optimizing the timing of when intelligent selection suggestions are provided within a VR/AR environment. In one example, the systems and methods described herein determine a probability that a potential action within a user interface is an intended action; quantify, over a period of time, a value of suggesting the potential action within the user interface; select a time at which to suggest the potential action based on the quantified value over the period of time; and suggest the potential action within the user interface at the selected time. Various other methods, systems, and computer-readable media are also disclosed.

Optimizing the timing of intelligent facilitation

The disclosed computer-implemented method may include capturing, by a computing device, a current position of a wearable device and a current position of user eye gaze at a current time. The method may also include determining that a user eye gaze is at a fixation position based on a speed of gaze movement. Additionally, the method may include calculating a direction of cursor movement by comparing a current cursor position with a previous cursor position at a previous time. Furthermore, the method may include calculating a likelihood of the fixation position being a target cursor position based on a difference between the direction of cursor movement and a direction from the current cursor position to the fixation position. Finally, the method may include increasing a speed of cursor movement toward the fixation position based on the likelihood. Various other methods, systems, and computer-readable media are also disclosed.

Systems and methods for gaze-assisted gesture control

In particular embodiments, a computing system may receive an image comprising one or more virtual elements associated with a virtual environment and one or more real-world elements associated with a real-world environment. The system may determine a first metric and a second metric indicative of a measure of clutter in the virtual environment and the real-world environment, respectively. The system may determine gaze features associated with a user based on a user activity and predict, using a machine learning model, a reaction time of the user based on the gaze features. The system may determine a third metric indicative of the measure of clutter in the image based on predicted reaction time. The system may compute an overall clutter metric based on the first, second, and third metrics. The system may perform one or more actions to manage the clutter in the image based on the overall clutter metric.

Managing visual clutter in artificial reality environments

The disclosed computer-implemented method may include (1) receiving information associated with a real-time output of a recognition model adapted to recognize at least one behavior of a user, (2) determining, based on the information, a level of uncertainty associated with the real-time output, (3) modulating at least one attribute of feedback based on the level of uncertainty, and (4) presenting the feedback to the user substantially contemporaneous with the real-time output of the recognition model. Various other methods, systems, and computer-readable media are also disclosed.

Systems and methods for communicating recognition-model uncertainty to users

Various aspects of the subject technology relate to systems, methods, and machine-readable media for navigating through a shared artificial reality environment. Various aspects may include receiving an indication of a virtual object in the shared artificial reality environment. Aspects may also include receiving an input gesture indicative of a navigation command associated with the virtual object. Aspects may also include determining at least one type of the input gesture comprising flexion and extension, pronation and supination, or radial and ulnar. Aspects may also include determining a control method. Aspects may include determining a scrolling parameter. Aspects may include identifying the navigation command based on the type of the input gesture, the control method, and the scrolling parameter. Aspects may include applying the navigation command to the virtual object.

Scrolling and navigation in virtual reality

An electronic device tracks, for a user performing a target acquisition movement within a 3D space, movement parameters of a plurality of input devices of the user. The electronic device predicts, for the user, a region of interest within the 3D space, based on the movement parameters. The region of interest includes a plurality of targets in close proximity. The electronic device predicts an endpoint of the target acquisition movement, within the region of interest. In some embodiments, the plurality of input devices includes an eye tracking input device, each input device corresponds to a predefined input device type, and the movement parameters include gaze data from the eye tracking input device. In some embodiments, input devices includes an eye tracking input device, a head-mounted display, and a hand-held controller, and the user's eye, hand, and head movements are coordinated.

Multimodal kinematic template matching and regression modeling for ray pointing prediction in virtual reality

The disclosed computer-implemented method may include detecting, by a computing system, a gesture that appears to be intended to trigger a response by the computing system, identifying, by the computing system, a context in which the gesture was performed, and adjusting, based at least on the context in which the gesture was performed, a threshold for determining whether to trigger the response to the gesture in a manner that causes the computing system to perform an action that is based on the detected gesture.

Systems and methods for adaptive input thresholding

An electronic device tracks, for a user performing a target acquisition movement within a 3D space, movement parameters of a plurality of input devices of the user. The electronic device predicts, for the user, a region of interest within the 3D space, using a regression model, based on the movement parameters. The region of interest includes a plurality of targets in close proximity. The electronic device predicts an endpoint of the target acquisition movement, within the region of interest, using a pointer facilitation technique. In some embodiments, the plurality of input devices includes an eye tracking input device, each input device corresponds to a predefined input device type, and the movement parameters include gaze data from the eye tracking input device. In some embodiments, input devices includes an eye tracking input device, a head-mounted display, and a hand-held controller, and the user's eye, hand, and head movements are coordinated.

Growing community of inventors

Seattle, WA, United States of America

Tanya Renee Jonker