Meta Platforms, Inc.

Etienne Pinchon

Jasper Stevens

Jonathan Ravasz

Robert David Ellis

Adam Tibor Varga

Jennifer Lynn Spurlock

Barrett Fox

Simon Charles Tickner

Kyle Eric Sorge-Toomey

Jonah Jones

Evgenii Krivoruchko

Wai Leong Chak

Roger Ibars Martinez

Chris Rojas

Gerrit Hendrik Hofmeester

Bojan Wilytsch

Christopher Alan Baker

An artificial reality system is described that renders, presents, and controls user interface elements within an artificial reality environment, and performs actions in response to one or more detected gestures of the user. The artificial reality system can include a menu that can be activated and interacted with using one hand. In response to detecting a menu activation gesture performed using one hand, the artificial reality system can cause a menu to be rendered. A menu sliding gesture (e.g., horizontal motion) of the hand can be used to cause a slidably engageable user interface (UI) element to move along a horizontal dimension of the menu while horizontal positioning of the UI menu is held constant. Motion of the hand orthogonal to the menu sliding gesture (e.g., non-horizontal motion) can cause the menu to be repositioned. The implementation of the artificial reality system does require use of both hands or use of other input devices in order to interact with the artificial reality system.

Artificial reality system having a sliding menu

An artificial reality system is described that renders, presents, and controls user interface elements within an artificial reality environment, and performs actions in response to one or more detected gestures of the user. In one example, an artificial reality system comprises an image capture device configured to capture image data representative of a physical environment; a head-mounted display (HMD) configured to output artificial reality content; a gesture detector configured to identify, from the image data, a gesture comprising a motion of two fingers from a hand to form a pinching configuration and a subsequent pulling motion while in the pinching configuration; a user interface (UI) engine configured to generate a UI input element in response to identifying the gesture; and a rendering engine configured to render the UI input element as an overlay to at least some of the artificial reality content.

Detecting input in artificial reality systems based on a pinch and pull gesture

Methods and systems described herein are directed to viewpoint coordination of multiple users on artificial reality collaboration models. Multiple users can collaborate on 3D models in an artificial reality environment. Attention indicators can indicate to the other users where a user is looking or where the user is pointing. In some implementations, there is a central model that each user sees from her own perspective (e.g., in the center of a virtual table all users are seated around) and/or instances of the model that only an individual user sees or can manipulate. The user's attention indicators can be based on where the user is looking or pointing on the model instances. Additionally, attention indicators can be translated from a controlling user's perspective to a coordinate system defined by the model, which allows the indicators to be then presented in that model coordinate system for all users.

Viewpoint coordination on artificial reality models

The present technology relates to artificial reality systems. Such systems provide projections a user can create to specify object interactions. For example, when a user wishes to interact with an object outside her immediate reach, she can use a projection to select, move, or otherwise interact with the distant object. The present technology also includes object selection techniques for identifying and disambiguating between objects, allowing a user to select objects both near and distant from the user. Yet further aspects of the present technology include techniques for interpreting various bimanual (two-handed) gestures for interacting with objects. The present technology further includes a model for differentiating between global and local modes for, e.g., providing different input modalities or interpretations of user gestures.

Automatic projection type selection in an artificial reality environment

Embodiments described herein disclose methods and systems directed to locomotion in virtual reality (VR) based on hand gestures of a user. In some implementations, the user can navigate between locations using hand gestures that trigger teleportation. In other implementations, the user can separately control forward/backward movement and the direction orientation the user is facing. The separate control can either be with one hand when making different gestures or by using different hands to control movement and orientation. In some implementations, a dragging gesture by the user is interpreted by the VR system to trigger movement. In this way, the user can turn in place without moving forward, move forward without turning, or can move forward and turn, while controlling speed, all with a single gesture.

Virtual reality locomotion via hand gesture

An artificial reality system is described that renders, presents, and controls user interface elements within an artificial reality environment, and performs actions in response to one or more detected gestures of the user. In one example, an artificial reality system comprises a head-mounted display configured to output artificial reality content; a stylus; a stylus action detector configured to detect movement of the stylus, detect a stylus selection action, and after detecting the stylus selection action, detect further movement of the stylus; a UI engine configured to generate stylus movement content in response to detecting movement of the stylus, and generate a UI input element in response to detecting the stylus selection action; and a rendering engine configured to render the stylus movement content and the UI input element as overlays to the artificial reality content, and update the stylus movement content based on the further movement of the stylus.

Detecting input using a stylus in artificial reality systems based on a stylus movement after a stylus selection action

Embodiments described herein disclose methods and systems directed to input mode selection in artificial reality. In some implementations, various input modes enable a user to perform precise interactions with a target object without occluding the target object. Some input modes can include rays that extend along a line that intersects an origin point, a control point, and an interaction point. An interaction model can specify when the system switches between input modes, such as modes based solely on gaze, using long or short ray input, or with direct interaction between the user's hand(s) and objects. These transitions can be performed by evaluating rules that take context factors such as whether a user's hands are in view of the user, what posture the hands are in, whether a target object is selected, and whether a target object is within a threshold distance from the user.

Artificial reality multi-modal input switching model

Projection casting in virtual environments

The present embodiments relate to generating 3D objects in an artificial reality environment and collapsing 3D objects into 2D images representing the 3D objects. Users operating extra reality (XR) devices controlling the artificial reality environment can collaboratively create or modify content in the artificial reality environment, using real-world creation objects (such as a hand of the user or a pen held by the user) to create 3D objects. In response to a user triggering a collapse for a 3D object, a 2D image of the 3D object can created, from the user's perspective. The 2D image can replace the 3D object in the artificial reality environment. Presenting the 2D image can reduce the amount of data, processing resources, and power needed to provide the artificial reality environment while also reducing clutter and cognitive load on the user.

Collapsing of 3D objects to 2D images in an artificial reality environment

Global and local mode virtual object interactions

A progressive display system can compute a virtual distance between a user and virtual objects. The virtual distance can be based on: a distance between the user and an object, a viewing angle of the object, and/or a footprint of the object in a field of view. The progressive display system can determine where the virtual distance falls in a sequence of distance ranges that correspond to levels of detail. Using a mapping between content sets for the object and levels of detail that correspond to distance ranges, the progressive display system can select content sets to display in relation to the object. As the user moves, the virtual distance will move across thresholds bounding the distance ranges. This causes the progressive display system to select and display other content sets for the distance range in which the current virtual distance falls.

Progressive display of virtual objects

An artificial reality system is described that renders, presents, and controls user interface elements within an artificial reality environment, and performs actions in response to one or more detected gestures of the user. The artificial reality system includes an image capture device configured to capture image data representative of a physical environment, a head-mounted display (HMD) configured to output artificial reality content, a gesture detector configured to identify, from the image data, a gesture comprising a configuration of a hand that is substantially stationary for at least a threshold period of time and positioned such that a thumb of the hand and at least one other finger form approximately a circle or approximately a circular segment, a user interface (UI) engine to generate a UI element in response to the identified gesture, and a rendering engine to render the UI element as an overlay to the artificial reality content.

Artificial reality systems with hand gesture-contained content window

A hand interaction system can use a three-state model to differentiate between normal hand movements, such as reaching for an object, and hand input gestures. The three-state model can specify a sequence of states including: 1) a neutral state, 2) a tracking state, and 3) an active state. In the neutral state, the hand interaction system monitors for a gesture signaling a transition to the tracking state but does not otherwise interpret a gesture corresponding to the active state as input. Once a gesture causes a transition to the intermediate tracking state, the hand interaction system can recognize a further active state transition gesture, allowing active state interaction. Thus, the monitoring for the intermediate tracking state provides a gating mechanism, making it less likely that the hand interaction system will interpret hand movements as input when not so intended by the user.

Growing community of inventors

London, United Kingdom

Etienne Pinchon