University of Minnesota

Apple Inc.

Stergios I Roumeliotis

Kejian J Wu

Chao X Guo

Oleg Naroditsky

Anastasios I Mourikis

Esha D Nerurkar

Dimitrios G Kottas

Joel Hesch

Georgios Georgiou

Oriel Bergig

Kuen-Han Lin

Andriy Grygorenko

Alex Flint

Christopher P Broaddus

Shuntaro Yamazaki

Daniel C Byrnes

Nicholas C Soldner

Qing He

Abdelhamid Dine

Tianheng Wang

Kourosh Sartipi

Tong Ke

Joseph Cheung

Jose A Rios

Connie Wu

Ravi Teja Sukhavasi

Ryan C DuToit

Methods for performing a calibration process for outward-facing cameras on devices such as head-mounted display devices are disclosed. Extrinsic parameters of the camera are first estimated using inputs to the calibration process such as information from an inertial measurement unit and points of interest within images captured by the camera that are tracked with time. Then, extrinsic and intrinsic parameters are concurrently determined in an optimization problem such that updated values of said parameters may be stored and used by applications that run on the device and make use of the camera. The calibration process may be extended to concurrently calibrate multiple cameras based, at least in part, on information from the inertial measurement unit that is local to the camera.

Methods for infield camera calibrations

A vision-aided inertial navigation system (VINS) comprises an image source for producing image data along a trajectory. The VINS further comprises an inertial measurement unit (IMU) configured to produce IMU data indicative of motion of the VINS and an odometry unit configured to produce odometry data. The VINS further comprises a processor configured to compute, based on the image data, the IMU data, and the odometry data, state estimates for a position and orientation of the VINS for poses of the VINS along the trajectory. The processor maintains a state vector having states for a position and orientation of the VINS and positions within the environment for observed features for a sliding window of poses. The processor applies a sliding window filter to compute, based on the odometry data, constraints between the poses within the sliding window and compute, based on the constraints, the state estimates.

Vision-aided inertial navigation system for ground vehicle localization

Various implementations disclosed herein include devices, systems, and methods that localize a device based on detecting planes in depth data acquired by the device. For example, an example process may include detecting first plane data in first sensor data acquired by a sensor at a first viewpoint location in a physical environment, detecting second plane data in second sensor data acquired by the sensor at a second viewpoint location in the physical environment, determining that the first plane data and the second plane data correspond to a same plane based on comparing the first plane data with the second plane data, and determining a spatial transformation between the first viewpoint location and the second viewpoint location based on the first plane data and the second plane data.

Plane extraction and localization

Vision-aided inertial navigation techniques are described. In one example, a vision-aided inertial navigation system (VINS) comprises an image source to produce image data at a first set of time instances along a trajectory within a three-dimensional (3D) environment, wherein the image data captures features within the 3D environment at each of the first time instances. An inertial measurement unit (IMU) to produce IMU data for the VINS along the trajectory at a second set of time instances that is misaligned with the first set of time instances, wherein the IMU data indicates a motion of the VINS along the trajectory. A processing unit comprising an estimator that processes the IMU data and the image data to compute state estimates for 3D poses of the IMU at each of the first set of time instances and 3D poses of the image source at each of the second set of time instances along the trajectory. The estimator computes each of the poses for the image source as a linear interpolation from a subset of the poses for the IMU along the trajectory.

Efficient vision-aided inertial navigation using a rolling-shutter camera with inaccurate timestamps

Methods for performing a camera calibration process for outward-facing cameras on devices such as head-mounted display devices are disclosed. Using cameras with overlapping fields of view, relative rotational parameters of the cameras with respect to one another may be determined using an optimization technique such as a two-view bundle adjustment algorithm. A statistical analysis of the relative rotational parameters of the cameras, determined for a plurality of moments in time, may then be made to provide updated relative rotational parameters for recalibration of the cameras. A camera calibration process, such as those disclosed, may not depend on tracking points of interest over multiple moments in time, but rather on a convergence of the relative rotational parameters determined for respective moments in time.

Methods for camera calibration via bundle adjustment

A processing unit of an electronic device may transmit a synchronization pulse to each of a first plurality of motion sensors (e.g., Inertial Measurement Units or IMUs) communicatively coupled to the processing unit, wherein the synchronization pulse transmitted to each motion sensor of the first plurality of motion sensors is assigned an offset time amount by the processing unit. The processing unit then obtains motion information samples from each of the first plurality of motion sensors, e.g., in an order corresponding to the offset time amounts assigned to the respective motion sensors, and then performs a processing operation based upon the obtained motion information samples. In some embodiments, each motion sensor may be assigned an equal offset time amount. In other embodiments, the offset time amount assigned to each motion sensor may be individually determined, e.g., based on one or more conditions, settings, or preferences related to the electronic device.

Latency reduction architecture using multiple motion sensors

Implementations of the subject technology provide extended reality display devices that can be used on and/or off of a moving platform. Systems and methods are disclosed for separating out the motion of the moving platform from other motions of the device so that virtual content can be displayed without erroneous motions caused by the motion of the moving platform. The subject technology can provide extended reality settings on any suitable moveable platform such as in a car, a watercraft, an aircraft, a train, or any other vehicle.

Inertial data management for extended reality for moving platforms

A vision-aided inertial navigation system (VINS) implements a square-root multi-state constraint Kalman filter (SR-MSCKF) for navigation. In one example, a processor of a VINS receives image data and motion data for a plurality of poses of a frame of reference along a trajectory. The processor executes an Extended Kalman Filter (EKF)-based estimator to compute estimates for a position and orientation for each of the plurality of poses of the frame of reference along the trajectory. For features observed from multiple poses along the trajectory, the estimator computes constraints that geometrically relate the multiple poses of the respective feature. Using the motion data and the computed constraints, the estimator computes state estimates for the position and orientation of the frame of reference. Further, the estimator determines uncertainty data for the state estimates and maintains the uncertainty data as a square root factor of a covariance matrix.

Square-root multi-state constraint Kalman filter for vision-aided inertial navigation system

Efficient vision-aided inertial navigation using a rolling-shutter camera

Localization and navigation systems and techniques are described. An electronic device comprises a processor configured to maintain a state vector storing estimates for a position of the electronic device at poses along a trajectory within an environment along with estimates for positions for one or more features within the environment. The processor computes, from the image data, one or more constraints based on features observed from multiple poses of the electronic device along the trajectory, and computes updated state estimates for the position of the electronic device in accordance with the motion data and the one or more computed constraints without computing updated state estimates for the features for which the one or more constraints were computed.

Vision-aided inertial navigation

A vision-aided inertial navigation system comprises an image source to produce image data for poses of reference frames along a trajectory, a motion sensor configured to provide motion data of the reference frames, and a hardware-based processor configured to compute estimates for a position and orientation of the reference frames for the poses. The processor executes a square-root inverse Schmidt-Kalman Filter (SR-ISF)-based estimator to compute, for features observed from poses along the trajectory, constraints that geometrically relate the poses from which the respective feature was observed. The estimator determines, in accordance with the motion data and the computed constraints, state estimates for position and orientation of reference frames for poses along the trajectory and computes positions of the features that were each observed within the environment. Further, the estimator determines uncertainty data for the state estimates and maintains the uncertainty data as a square root factor of a Hessian matrix.

Square root inverse Schmidt-Kalman filters for vision-aided inertial navigation and mapping

Localization and navigation systems and techniques are described. An electronic device comprises a processor configured to apply an extended Kalman filter (EKF) as the electronic device traverses the trajectory. The extended Kalman filter is configured to maintain a state vector storing estimates for a position of the electronic device at poses along a trajectory within an environment along with estimates for positions for one or more features within the environment. The EKF computes constraints based on features observed from multiple poses along the trajectory, and updates, in accordance with motion data and the one or more computed constraints, the estimates within the state vector of the extended Kalman filter while excluding, from the state vector, state estimates for positions within the environment for the features that were observed from the multiple poses and for which the constraints were computed.

Growing community of inventors

Saint Paul, MN, United States of America

Stergios I Roumeliotis