Snap Inc.

Daniel Wolf

Matthias Kalkgruber

Georg Halmetschlager-Funek

Jakob Zillner

Olha Borys

Thomas Muttenthaler

Daniel Wagner

Kai Zhou

Georgios Evangelidis

Erick Mendez Mendez

Bernhard Jung

Gerald Nilles

John James Robertson

Sumant Milind Hanumante

Ozi Egri

Dmitry Ryuma

Jeroen Diederik Hol

Branislav Micusik

Bo Ding

Edward Lee Kim-Koon

Alexander Sourov

Anton Kondratenko

Thomas Faeulhammer

Nikolaj Kuntner

Ilteris Kaan Canberk

Brian Fulkerson

Georgios Papandreou

Daniel Colascione

Tiago Miguel Pereira Torres

Markus Diem

Erick Mendez

Jan Bajana

Márton Gergely Kajtár

Balázs Tóth

In examples described herein, a sensor external to an extended reality (XR) device is connected to an extremity of a user of the XR device. The external sensor is communicatively coupled to the XR device. The XR device captures image data comprising one or more images of the extremity of the user. The XR device accesses external tracking data generated by the external sensor. A forecast of a pose of the extremity is generated based on the image data and the external tracking data. The forecast may be used for tracking of the extremity or to render virtual content for presentation to the user.

Body-mounted sensor for improved pose forecasting

Examples in the present disclosure relate to hand chirality estimation. Tracking data captured by one or more sensors associated with an extended reality (XR) device is processed to determine positions of a plurality of joints of a hand of a person. A reference vector is generated based on a first subset of the positions. The first subset of the positions includes positions of at least two metacarpophalangeal joints. A plurality of bending angles is determined based on at least a second subset of the positions. Each bending angle represents an angle between a respective pair of articulating bones that is measured in relation to the reference vector. An estimated chirality of the hand is identified based on the plurality of bending angles. Operation of the XR device is controlled using the estimated chirality of the hand.

Hand chirality estimation for extended reality tracking

Examples disclosed herein relate to the use of shared pose data in extended reality (XR) tracking. A communication link is established between a first XR device and a second XR device. The second XR device is worn by a user. The first XR device receives pose data of the second XR device via the communication link and captures an image of the user. The user is identified based on the image and the pose data.

Extended reality tracking using shared pose data

A hand-tracking input pipeline dimming system for an AR system is provided. The AR system deactivates the hand-tracking input pipeline and places a camera component of the hand-tracking input pipeline in a limited operational mode. The AR system uses the camera component to detect initiation of a gesture by a user of the AR system and in response to detecting the initiation of the gesture, the AR system activates the hand-tracking input pipeline and places the camera component in a fully operational mode.

Hand-tracking pipeline dimming

Visual-inertial tracking of an eyewear device using a rolling shutter camera(s). The device includes a position determining system. Visual-inertial tracking is implemented by sensing motion of the device. An initial pose is obtained for a rolling shutter camera and an image of an environment is captured. The image includes feature points captured at a particular capture time. A number of poses for the rolling shutter camera is computed based on the initial pose and sensed movement of the device. The number of computed poses is responsive to the sensed movement of the mobile device. A computed pose is selected for each feature point in the image by matching the particular capture time for the feature point to the particular computed time for the computed pose. The position of the mobile device is determined within the environment using the feature points and the selected computed poses for the feature points.

Visual-inertial tracking using rolling shutter cameras

A method for limiting motion blur in a visual tracking system is described. In one aspect, the method includes accessing a first image generated by an optical sensor of the visual tracking system, identifying camera operating parameters of the optical sensor for the first image, determining a motion of the optical sensor for the first image, determining a motion blur level of the first image based on the camera operating parameters of the optical sensor and the motion of the optical sensor, and adjusting the camera operating parameters of the optical sensor based on the motion blur level.

Dynamic adjustment of exposure and ISO to limit motion blur

Bending data is used to facilitate tracking operations of an extended reality (XR) device, such as hand tracking or other object tracking operations. The XR device obtains bending data indicative of bending of the XR device to accommodate a body part of a user wearing the XR device. The XR device determines, based on the bending data, whether to use previously identified biometric data in a tracking operation. A mode of the XR device is selected responsive to determining whether to use the previously identified biometric data. The selected mode is used to initialize the tracking operation. The selected mode may be a first mode in which the previously identified biometric data is used in the tracking operation or a second mode in which the previously identified biometric data is not used in the tracking operation.

Bending-assisted calibration for extended reality tracking

Visual-inertial tracking of an eyewear device using sensors. The eyewear device monitors the sensors of a visual inertial odometry system (VIOS) that provide input for determining a position of the device within its environment. The eyewear device determines the status of the VIOS based information from the sensors and adjusts the plurality of sensors (e.g., by turning on/off sensors, changing the sampling rate, of a combination thereof) based on the determined status. The eyewear device then determines the position of the eyewear device within the environment using the adjusted plurality of sensors.

Dynamic sensor selection for visual inertial odometry systems

A method for aligning coordinate systems of user devices in an augmented reality system using somatic points of a user's hand as alignment markers. Images captured from multiple user devices are used to align the reference coordinate systems of the user devices to a common reference coordinate system. In some examples, user devices capture images of a hand of a user and use object recognition to identify somatic points as alignment markers. The somatic points of a user device are translated to a common reference coordinate system determined by another user device.

Shared augmented reality eyewear device with hand tracking alignment

A method for carving a 3D space using hands tracking is described. In one aspect, a method includes accessing a first frame from a camera of a display device, tracking, using a hand tracking algorithm operating at the display device, hand pixels corresponding to one or more user hands depicted in the first frame, detecting, using a sensor of the display device, depths of the hand pixels, identifying a 3D region based on the depths of the hand pixels, and applying a 3D reconstruction engine to the 3D region.

3D space carving using hands for object capture

A method for calibrating a visual-inertial tracking system is described. A device operates the visual-inertial tracking system without receiving a tracking request from a virtual object display application. In response to operating the visual-inertial tracking system, the device accesses sensor data from sensors at the device. The device identifies, based on the sensor data, a first calibration parameter value of the visual-inertial tracking system and stores the first calibration parameter value. The system detects a tracking request from the virtual object display application. In response to the tracking request, the system accesses the first calibration parameter value and determines a second calibration parameter value from the first calibration parameter value.

Periodic parameter estimation for visual-inertial tracking systems

A method for mitigating motion blur in a visual-inertial tracking system is described. In one aspect, the method includes accessing a first image generated by an optical sensor of the visual tracking system, accessing a second image generated by the optical sensor of the visual tracking system, the second image following the first image, determining a first motion blur level of the first image, determining a second motion blur level of the second image, identifying a scale change between the first image and the second image, determining a first optimal scale level for the first image based on the first motion blur level and the scale change, and determining a second optimal scale level for the second image based on the second motion blur level and the scale change.

Direct scale level selection for multilevel feature tracking under motion blur

A method for mitigating motion blur in a visual tracking system is described. In one aspect, a method for selective motion blur mitigation in a visual tracking system includes accessing a first image generated by an optical sensor of the visual tracking system, identifying camera operating parameters of the optical sensor during the optical sensor generating the first image, determining a motion of the optical sensor during the optical sensor generating the first image, determining a motion blur level of the first image based on the camera operating parameters of the optical sensor and the motion of the optical sensor, and determining whether to downscale the first image using a pyramid computation algorithm based on the motion blur level.

Selective image pyramid computation for motion blur mitigation in visual-inertial tracking

Disclosed are systems, methods, and non-transitory computer-readable media for misaligned vantage point mitigation for computer stereo vision. A misaligned vantage point mitigation system determines whether vantage points of the optical sensors are misaligned from an expected vantage point and, if so, determines an adjustment variable to mitigate the misalignment based on the location of matching features identified in images captured by both optical sensors.

Misaligned vantage point mitigation for computer stereo vision

Devices and methods for dynamic power configuration (e.g., reduction) for thermal management (e.g., mitigation) in a wearable electronic device such as an eyewear device. The wearable electronic device monitors its temperature and, responsive to the temperature, configures the services it provides to operate in different modes for thermal mitigation (e.g., to prevent overheating). For example, based on temperature, the wearable electronic device adjusts sensors (e.g., turns cameras on or off, changes the sampling rate, or a combination thereof) and adjusts display components (e.g., adjusted rate at which a graphical processing unit generates images and a visual display is updated). This enables the wearable electronic device to consume less power when temperatures are too high in order to provide thermal mitigation.

Growing community of inventors

Modling, Austria

Daniel Wolf