Facebook Technologies, LLC

Meta Platforms Technologies, LLC

Christian Forster

Chengyuan Yan

Tsz Ho Yu

Oskar Linde

In one embodiment, a method includes capturing, using one or more cameras implemented in a wearable device worn by a user, a first image depicting at least a part of a hand of the user holding a controller in an environment, identifying one or more features from the first image to estimate a pose of the hand of the user, estimating a first pose of the controller based on the pose of the hand of the user and an estimated grip that defines a relative pose between the hand of the user and the controller, receiving IMU data of the controller, and estimating a second pose of the controller by updating the first pose of the controller using the IMU data of the controller. The method utilizes multiple data sources to track the controller under various conditions of the environment to provide an accurate controller tracking consistently.

Systems and methods for object tracking using fused data

In one embodiment, a computing system is configured to, during a first tracking session, detect first landmarks in a first image of the environment surrounding a user, and determine a first location of the user by comparing detected first landmarks to a landmark database. During a second tracking session, the computing system captures motion data and estimates a second location of the user based on the motion data and first user location. Based on the motion data and first user location, the computing system detects landmarks in a second image at a second location. The system accesses expected landmarks from the landmark database visible at the second location and determines the estimated second location of the user is inaccurate by comparing the expected landmarks with the second landmarks. The computing system re-localizes the user by comparing the landmarks in the landmark database and third landmarks in a third image.

System and method for improving localization and object tracking

The disclosed computer-implemented method for offloading image-based tracking operations from a general processing unit to a hardware accelerator unit may include (1) sending imaging data from an imaging device to a hardware accelerator unit, and (2) directing the hardware accelerator unit to generate a multi-scale representation of the imaging data sent from the imaging device, (3) preparing a set of input data for a set of image-based tracking operations, and (4) directing the hardware accelerator unit to execute the set of image-based tracking operations using the generated multi-scale representation of the imaging data and the prepared set of input data. Various other methods, systems, and computer-readable media are also disclosed.

Systems and methods for offloading image-based tracking operations from a general processing unit to a hardware accelerator unit

In one embodiment, a computing system is configured to track objects in an environment or localize a user device. For example, the system accesses an image of an environment captured from a viewpoint. Based on the image, the system detects landmarks that are associated with objects in the environment and identifies expected landmarks that are expected to be observable from the viewpoint using a landmark database. The system determines that at least one of the expected landmarks is currently unobservable in the environment by comparing the expected landmarks with the detected landmarks. By accessing semantic information associated with the at least one expected landmark, the system updates the landmark database based on a determination that the semantic information satisfies predetermined criteria and removes the at least one expected landmark from the landmark database. The system performs object tracking, object mapping, or re-localization within the environment using the updated landmark database.

Growing community of inventors

Baar, Switzerland

Christian Forster