Meta Platforms Technologies, LLC

Facebook Technologies, LLC

Facebook, Inc.

Meta Platforms, Inc.

Chengyuan Yan

Ke Huo

Anush Mohan

Adrian Brian Ratter

Boyang Zhang

Paul Austin Buckley

Chengyuan Lin

Amrutha Hakkare Arunachala

Jack Hood Profit, Jr

Sheng Shen

Christian Forster

Joseph Davis Greer

Tsz Ho Yu

Ganesh Venkatesh

Oskar Linde

Willy Huang

Szeyin Lee

John Jacob Blakeley

Christopher E Balmes

Amy Lawson Bearman

Yiming Pu

Matthew Dan Feiszli

Alireza Dirafzoon

Gabrielle Catherine Moskey

Anurag Sharma

Babak Damavandi

Guangqiang Dong

Yinglin Li

Qian Chen

Jiwen Ren

The various implementations described herein include methods and systems for providing input capabilities at various fidelity levels. In one aspect, a method includes receiving, from an application, a request identifying an input capability for making an input operation available within the application. The method further includes, in response to receiving the request: identifying techniques that the artificial-reality system can use to make the requested input capability available to the application using data from one or more devices; selecting a first technique for making the requested input capability available to the application; and using the first technique to provide to the application data to allow for performance of the requested input capability.

Hardware-agnostic input framework for providing input capabilities at various fidelity levels, and systems and methods of use thereof

Methods of interpreting in-air hand gestures based on orientations of a wrist-wearable device are provided. The method includes receiving, from sensors of a wrist-wearable device, data associated with performance of an in-air hand gesture during a first period of time. The method includes, when determining the data indicates that the wrist-wearable device had a first orientation when the in-air hand gesture was performed during the first period of time, causing performance of a first operation. The method includes receiving, from sensors of the wrist-wearable device, new data associated with performance of the in-air hand gesture during a second period of time, the second period of time being after the first period of time. The method includes, when determining that the new data indicates that the wrist-wearable device had a second orientation when the in-air hand gesture was performed during the second period of time, causing performance of a second operation.

Systems for interpreting thumb movements of in-air hand gestures for controlling user interfaces based on spatial orientations of a user's hand, and method of use thereof

In one embodiment, a method includes capturing, using one or more cameras implemented in a wearable device worn by a user, a first image depicting at least a part of a hand of the user holding a controller in an environment, identifying one or more features from the first image to estimate a pose of the hand of the user, estimating a first pose of the controller based on the pose of the hand of the user and an estimated grip that defines a relative pose between the hand of the user and the controller, receiving IMU data of the controller, and estimating a second pose of the controller by updating the first pose of the controller using the IMU data of the controller. The method utilizes multiple data sources to track the controller under various conditions of the environment to provide an accurate controller tracking consistently.

Systems and methods for object tracking using fused data

A method of interpreting a digit-to-digit gesture based on roll values for a wrist-wearable device is disclosed. The method includes, receiving an indication at a first point in time that a user donning the wrist-wearable device is providing a digit-to-digit gesture in which one of the user's digits touches another of the user's digits without contacting the display of the wearable device. The method further includes, in accordance with a determination that the digit-to-digit gesture is provided while data indicates that the wrist-wearable device has a first roll value, causing a target device in communication with the wearable device to perform a first input command and receiving another indication at a second point in time that is after the first point in time that the user is providing the digit-to-digit gesture again, causing the target device to perform a second input command that is distinct from the first input command.

Systems for interpreting a digit-to-digit gesture by a user differently based on roll values of a wrist-wearable device worn by the user, and methods of use thereof

In one embodiment, a method includes rendering a first output image comprising one or more augmented-reality (AR) objects for displays of an AR rendering device of an AR system associated with a first user. The method further includes accessing sensor signals associated with the first user. The one or more sensor signals may be captured by sensors of the AR system. The method further includes detecting a change in a context of the first user with respect to a real-world environment based on the sensor signals. The method further includes rendering a second output image comprising the AR objects for the displays of the AR rendering device. One or more of the AR objects may be adapted based on the detected change in the context of the first user.

Dynamic content rendering based on context for AR and assistant systems

Systems and methods for providing a virtual space for multiple devices can include a first device having at least one sensor configured to acquire a spatial information of a physical space of the first device. The first device may include at least one processor configured to establish, according to the acquired spatial information, a virtual space corresponding to the physical space, that is accessible by a user of the first device via the first device. The at least one processor may further be configured to register a second device within the physical space, to allow a user of the second device to access the virtual space via the second device.

Systems and methods for maintaining virtual spaces

Systems and methods for maintaining a shared interactive environment include receiving, by a server, requests to register a first input device of a first user and a second input device of a second user with a shared interactive environment. The first input device may be for a first modality involving user input for an augmented reality (AR) environment, and the second input device may be for a second modality involving user input for a personal computer (PC) based virtual environment or a virtual reality (VR) environment. The server may register the first and second input device with the shared interactive environment. The server may receive inputs from a first adapter for the first modality and from a second adapter for the second modality. The inputs may be for the first and second user to use the shared interactive environment.

Systems and methods for a shared interactive environment

In one embodiment, a method includes receiving, from a controller, a data packet including (1) a plurality of samples each corresponding to measurements from a motion sensor and (2) a timestamp corresponding to a measurement time of one of the samples as measured by a clock of the controller; determining, based on the timestamp, an estimated measurement time relative to a local clock for each of the plurality of samples that is not associated with the timestamp; and converting each of the timestamp and the estimated measurement times to a corresponding synchronization time using a learned relationship relating the clock of the controller and the local clock. The learned relationship is iteratively learned based on previously received data packets from the controller. The synchronization time associated with each of the plurality of samples represents an estimated time, relative to the local clock, at which the sample was measured.

Headset clock synchronization

The disclosed computer-implemented method may include tracking, using a low-order degree-of-freedom (DOF) mode, an orientation of a device based on input from an inertial measurement unit (IMU) of the device. The method may also include determining, using a magnetometer, that the device has entered a magnetic tracking volume defined by at least one magnet and in response to determining that the device has entered the magnetic tracking volume, transitioning from the low-order DOF mode to a high-order DOF mode that tracks a higher number of DOFs than the low-order DOF mode. The method may also include tracking, using the high-order DOF mode, the position and orientation of the device based on input from both the IMU and the magnetometer. Various other methods, systems, and computer-readable media are also disclosed.

Systems and methods for transitioning to higher order degree-of-freedom tracking

An artificial reality system is described includes a hand-held controller tracking sub-system having two components, a Field-of-View (FOV) tracker and a non-FOV tracker that applies specialized motion models when one or more of controllers are not trackable within the field of view. In particular, under typical operating conditions, the FOV tracker receives state data for a Head Mounted Display (HMD) and controller state data (velocity, acceleration etc.) of a controller to compute estimated poses for the controller. If the controller is trackable (e.g., within the field of view and not occluded), then the pose as computed by the FOV tracker is used and the non-FOV tracker is bypassed. If the controller is not trackable within the field of view and the controller state data meets activation conditions for one or more corner tracking cases, then the non-FOV tracker applies one or more of specialized motion models to compute a controller pose for the controller.

Multi-layered artificial reality controller pose tracking architecture having prioritized motion models

In one embodiment, a method includes receiving motion data from a motion sensor during a packet-transmission interval of a wireless protocol. The motion data corresponds to a first pre-determined number of samples measured at a first sampling frequency. Each sample is associated with a first timestamp corresponding to a measurement time of that sample during the packet-transmission interval. The method also includes converting the motion data to correspond to a second pre-determined number of samples. The second pre-determined number is fewer than the first pre-determined number. The method also includes determining a second timestamp for each of the second pre-determined number of samples. The second timestamps are within the packet-transmission interval and represent measurement times at a second sampling frequency that is lower than the first sampling frequency. The method also includes combining the converted motion data and the corresponding second timestamps into a first data packet.

Sensor data compression

Growing community of inventors

Fremont, CA, United States of America

Chengyuan Yan