Samsung Electronics Co., Ltd.

Meta Platforms Technologies, LLC

Aaron S Faucher

Andrew R McHugh

Edgar Charles Evangelista

Pol Pla I Conesa

Jaehyun Kim

Nathan Aschenbach

Daniel Rosas

Stephanie Kim Ahn

Chris Rojas

Bryan Sparks

Emron Jackson Henry

Johnathon Simmons

Sylvia Leung

Aspects of the present disclosure are directed to triggering virtual keyboard selections using multiple input modalities. An interface manager can display an interface, such as a virtual keyboard, to a user in an artificial reality environment. Implementations of the interface manager can track user eye gaze input and user hand input (e.g., hand or finger motion). The interface manager can resolve a character selection on the virtual keyboard according to the tracked user gaze input based on detection that the user's hand motion meets a trigger criteria. For example, the interface manager can: detect that the tracked user hand motion meets the trigger criteria at a given point in time; and resolve a selection from the virtual keyboard (e.g., selection of a displayed character) according to the tracked user gaze on the virtual keyboard at the given point in time.

Virtual selections using multiple input modalities

Aspects of the disclosure are directed to an interface for receiving input using multiple modalities in an artificial reality environment. The interface can be a virtual keyboard displayed in an artificial reality environment that includes characters arranged as elements. Implementations include an artificial reality device/system for displaying the artificial reality environment and receiving user input in a first modality, and a controller device for receiving user input in an additional input modality. For example, the artificial reality system can be configured to receive user gaze input as a first input modality and the controller device can be configured to receive input in a second modality, such as touch input received at a trackpad. An interface manager can process input in the multiple modalities to control an indicator on the virtual interface. The interface manager can also resolve character selections from the virtual interface according to the input.

Artificial reality input using multiple modalities

Virtual keyboard selections using multiple input modalities

One embodiment provides a method comprising determining a spatial relationship between an augmented reality (AR) device and a camera-equipped device. The AR device is worn by a user. The camera-equipped device is positioned within proximity of the user. The method comprises determining, based on the spatial relationship, minimum and maximum boundaries of a user volume encompassing a head and one or more extremities of the user. The minimum and maximum boundaries represent a reduced depth range that is narrower than a full depth range of a camera frustum of the camera-equipped device. The method further comprises encoding, based on the minimum and maximum boundaries, a depth image frame captured via the camera-equipped device, and transmitting the encoded depth image frame over a remote network connection. A reconstructed 3D image based in part on the encoded depth image frame is rendered by a different AR device.

Efficient encoding of depth data across devices

Extended reality interactions include capturing, with a first device, video of a first user and conveying same to a second, heterogeneous device. A 3D mesh is received by the first device from the second device for rendering an extended reality environment, which is simultaneously displayed on the second device. Video of a second user and pose transforms for compositing the video of the second user in the extended reality environment displayed on the first device is received. A view perspective of the video of the second user composited in the extended reality environment is based on the pose transforms. Input to the first device changes the view perspective. View perspective data is conveyed from the first device to the second device that causes a corresponding change in view perspective of the video of the first user composited in the extended reality environment simultaneously displayed on the second device.

Extended reality interaction in synchronous virtual spaces using heterogeneous devices

A calling method can include determining, with a device, a position of a nearby device in response to detecting a signal transmitted from the nearby device and capturing, with a camera of the device, an image of an area near the device. Responsive to identifying an image of a call candidate appearing within the image of the area, a position of the call candidate can be determined from the image. The position of the call candidate can be correlated with the position of the nearby device based on proximity. Information associated with the call candidate can be retrieved based on the correlating. Based on the information retrieved, a visual identifier token corresponding to the call candidate can be generated. The visual identifier token can be presented on a display of the device and can be used by the user to initiate a call between the device and the nearby device.

Augmented reality calling interface

An augmented reality (AR) device supporting an AR is provided. The AR device includes a display, a communication circuit, at least one processor operatively connected to the display and the communication circuit, and a memory operatively connected to the at least one processor. The memory stores instructions that, when executed, cause the at least one processor to establish a connection with a user device storing a contact application and a message application, through the communication circuit, detect that the message application is executed, and display a first graphic user interface (GUI) and at least one avatar, which is disposed at a location adjacent to the first GUI and corresponds to at least one contact associated with the contact application or the message application, through the display in the AR.

Electronic device for communicating in augmented reality and method thereof

One embodiment provides a method comprising determining a spatial relationship between an augmented reality (AR) device and a camera-equipped device. The AR device is worn by a user. The camera-equipped device is positioned within proximity of the user. The method further comprises determining a position of the user relative to a field of view of the camera-equipped device, and providing a representation of the user for display. The representation automatically switches between a real image of the user and a virtual avatar of the user based on the position of the user.

Growing community of inventors

San Mateo, CA, United States of America

Aaron S Faucher