Google Inc.

Luca Ballan

Chao X Guo

Qiyue Zhang

Mahesh Ramachandran

Konstantine Nicholas John Tsotsos

Fengtao Fan

Hongsheng Yu

Christopher Ross

Sazzadur Rahman

Ivo Duarte

José Carlos Maranhão Pascoal

Fei Han

Junyang Lu

Jie Zhang

Young Hwan Kim

Lucy Abramyan

Maria Nika Yocke

Joao Afonso

A tracking system and associated algorithms are disclosed that can provide a user with a continuous, reset-free augmented reality (AR) experience. When the user wears an AR headset equipped with a camera and an inertial measurement unit (IMU), motion sensor data from the IMU can be combined with image data from the camera to create a device pose, representing a position and an orientation of the headset relative to its environment. In some implementations, when a reset occurs, a six-degrees-of-freedom (6DoF) algorithm can be configured to support the pose until a re-initialization is completed. In some implementations, a neural network can be used to correct for IMU integration drifts in the 6DoF algorithm. In some implementations, the IMU-based 6DoF uses a neural network that exploits the device's past motion to infer its future motion.

Pose algorithm for continuous AR experiences

Techniques of providing continual augmented reality navigation include combining first and second 6DoF poses from, respectively, a localization from a map in a vicinity of a location to an image and from predicted 6DoF poses based on IMU data. The first 6DoF pose is based on a visual positioning system that uses map data from a vicinity of a location combined with an image from the world-facing camera of the AR smartglasses that provides 6DoF pose in world coordinates independent of the IMU. The second 6DoF pose is based on compensated rotational velocity and acceleration measurements derived from IMU intrinsic values.

Map-aided inertial odometry with neural network for augmented reality devices

Systems and techniques for persistent calibration of an electronic device configured to implement an extended reality (XR) system involve estimating and validating visual-inertial odometry (VIO) calibration parameters during an active XR session of the electronic device. Validating the estimated VIO calibration parameters involves performing a strict calibration qualification of the estimated VIO calibration parameters using a thresholding module, machine learning module, or both. An initial calibration qualification is performed concurrently with the strict calibration qualification based on VIO performance. If the estimated VIO calibration parameters pass the strict calibration qualification and initial calibration qualification, they are stored for use to calibrate the device in future XR sessions. Persistent calibration of time alignment between the inertial management unit and the image sensor of the electronic device is also performed during active XR sessions upon detection of time alignment issues.

Persistent calibration of extended reality systems

A method can include determining, by a head-mounted device based on first data received from a depth-sensing device included in the head-mounted device, that a first distance of the head-mounted device from a landmark is outside a completion range; outputting, based on the determination that the first distance of the head-mounted device from the landmark is outside the completion range from the landmark, an indication that a repetition has yet to be completed; determining, based on second data received from the depth-sensing device, that a second distance of head-mounted device from the landmark is within the completion range; and based on the determination that the second distance of the head-mounted device from the landmark is within the completion range, incrementing a repetition counter and outputting an indication that the repetition has been completed.

Tracking repetitions by head-mounted device based on distance

Techniques of providing continual augmented reality navigation include combining first and second 6DoF poses from, respectively, a localization from a map in a vicinity of a location to an image and from predicted 6DoF poses based on IMU data.

Growing community of inventors

San Jose, CA, United States of America

Luca Ballan