Snap Inc.

Meta Platforms, Inc.

Rpx Corporation

Georgios Evangelidis

Branislav Micusik

Daniel Wagner

Daniel Wolf

Ramzi Zahreddine

Markus Diem

Brevin Tilmon

Sizhuo Ma

Jordy Innocentius Ajanohoun

Matthew Penney

Ilteris Kaan Canberk

Sagi Katz

Sergey Tulyakov

Gurunandan Krishnan Gorumkonda

Jian Ren

Jennica Pounds

Ashwani Arya

Bernhard Jung

Erick Mendez Mendez

Nathan Litke

Daniel Colascione

Yang Wen

Jakob Zillner

Yicheng Wu

Jian Wang

Andrei Rybin

Ju Hu

Márton Gergely Kajtár

Yanyu Li

Jan Bajana

Ihor Tymchyshyn

Laura Rosalia Luidolt

Simon Schreiberhuber

Geng Yuan

Balázs Tóth

Vsevolod Konjahhin

Dmytro Siryk

A head-worn device system equipped with cameras, display devices, and processors uses stored instructions to facilitate interactions between an augmented reality (AR) device and an external display system. When executed, these instructions establish a communications link between the AR device and the external display. The system identifies the pose of the external display and receives user inputs from the AR device user, relating to interactions with virtual objects in a real-world setting. It also identifies the positions of viewers watching the external display. Based on the external display's pose and the viewer locations, the system generates display data for the virtual object, ensuring it appears correctly on the external display. This display data is then transmitted to the external display system, completing the interaction loop and enhancing the viewing experience for the audience.

Augmented reality (AR) content-sharing with AR-deprived audience

An energy-efficient adaptive 3D sensing system. The adaptive 3D sensing system includes one or more cameras and one or more projectors. The adaptive 3D sensing system captures images of a real-world scene using the one or more cameras and computes depth estimates and depth estimate confidence values for pixels of the images. The adaptive 3D sensing system computes an attention mask based on the one or more depth estimate confidence values and commands the one or more projectors to send a distributed laser beam into one or more areas of the real-world scene based on the attention mask. The adaptive 3D sensing system captures 3D sensing image data of the one or more areas of the real-world scene and generates 3D sensing data for the real-world scene based on the 3D sensing image data.

Energy-efficient adaptive 3D sensing

A method for aligning coordinate systems from separate augmented reality (AR) devices is described. In one aspect, the method includes generating predicted depths of a first point cloud by applying a pre-trained model to a first single image generated by a first monocular camera of a first augmented reality (AR) device, and first sparse 3D points generated by a first SLAM system at the first AR device, generating predicted depths of a second point cloud by applying the pre-trained model to a second single image generated by a second monocular camera of the second AR device, and second sparse 3D points generated by a second SLAM system at the second AR device, determining a relative pose between the first AR device and the second AR device by registering the first point cloud with the second point cloud.

Fast AR device pairing using depth predictions

A hand-tracking input pipeline dimming system for an AR system is provided. The AR system deactivates the hand-tracking input pipeline and places a camera component of the hand-tracking input pipeline in a limited operational mode. The AR system uses the camera component to detect initiation of a gesture by a user of the AR system and in response to detecting the initiation of the gesture, the AR system activates the hand-tracking input pipeline and places the camera component in a fully operational mode.

Hand-tracking pipeline dimming

A first extended reality (XR) device and a second XR device are colocated in an environment. The first XR device captures sensory data of a wearer of the second XR device. The sensory data is used to determine a time offset between a first clock of the first XR device and a second clock of the second XR device. The first clock and the second clock are synchronized based on the time offset and a shared coordinate system is established. The shared coordinate system enables alignment of virtual content that is simultaneously presented by the first XR device and the second XR device based on the synchronization of the first clock and the second clock.

Time synchronization for shared extended reality experiences

A method for aligning coordinate systems from separate handheld devices is described. In one aspect, the method includes accessing first pose data of a first handheld device, receiving second pose data of a second handheld device, detecting, from the first handheld device, hand-tracking data of a second user holding the second handheld device, and aligning a first coordinate system of the first handheld device with a second coordinate system of the second handheld device based on the first pose data, the second pose data, and the hand-tracking data of the second user holding the second handheld device.

Device-to-device collocated AR using hand tracking

Eyewear providing an interactive augmented reality experience between two eyewear devices by using alignment between respective 6DOF trajectories, also referred to herein as ego motion alignment. An eyewear device of user A and an eyewear device of user B track the eyewear device of the other user, or an object of the other user, such as on the user's face, to provide the collaborative AR experience. This enables sharing common three-dimensional content between multiple eyewear users without using or aligning the eyewear devices to common image content such as a marker, which is a more lightweight solution with reduced computational burden on a processor. An inertial measurement unit may also be used to align the eyewear devices.

Collaborative augmented reality eyewear with ego motion alignment

A method for aligning coordinate systems of user devices in an augmented reality system using somatic points of a user's hand as alignment markers. Images captured from multiple user devices are used to align the reference coordinate systems of the user devices to a common reference coordinate system. In some examples, user devices capture images of a hand of a user and use object recognition to identify somatic points as alignment markers. The somatic points of a user device are translated to a common reference coordinate system determined by another user device.

Shared augmented reality eyewear device with hand tracking alignment

A method for carving a 3D space using hands tracking is described. In one aspect, a method includes accessing a first frame from a camera of a display device, tracking, using a hand tracking algorithm operating at the display device, hand pixels corresponding to one or more user hands depicted in the first frame, detecting, using a sensor of the display device, depths of the hand pixels, identifying a 3D region based on the depths of the hand pixels, and applying a 3D reconstruction engine to the 3D region.

3D space carving using hands for object capture

A method for aligning coordinate systems from separate augmented reality (AR) devices is described. In one aspect, a first device accesses first pose data from a first Visual Inertial Odometry (VIO) system of the first device. A camera of the first device captures an image of a machine-readable code that is displayed on a display of a second device. The second device encodes the machine-readable code with second pose data from a second VIO system of the second device. The first device decodes the second pose data from the machine-readable code, and determines a relative pose between the first device and the second device based on the first pose data and the second pose data.

Device pairing using machine-readable optical label

Eyewear having unsynchronized rolling shutter (RS) cameras such that images produced by each camera are unaligned, and a state that includes velocity and gravity orientation in the eyewear's reference system is calculated. The state is not limited to these two parameters as other parameters such as the acceleration bias, the gyroscope bias, or both may be included. Math solvers are used such that processing time to calculate the velocity and gravity orientation are acceptable. Arranging the RS cameras in an unsynchronized configuration allows estimating the motion of the eyewear from just one stereo image pair and removes the requirement of possessing more images.

Eyewear having unsynchronized rolling shutter cameras

A vision transformer network having extremely low latency and usable on mobile devices, such as smart eyewear devices and other augmented reality (AR) and virtual reality (VR) devices. The transformer network processes an input image, and the network includes a convolution stem configured to patch embed the image. A first stack of stages including at least two stages of 4-Dimension (4D) metablocks (MBs) (MB) follow the convolution stem. A second stack of stages including at least two stages of 3-Dimension MBs (MB) follow the MBstages. Each of the MBstages and each of the MBstages include different layer configurations, and each of the MBstages and each of the MBstages include a token mixer. The MBstages each additionally include a multi-head self attention (MHSA) processing block.

Efficientformer vision transformer

Electronic eyewear device providing simplified audio source separation, also referred to as voice/sound unmixing, using alignment between respective device trajectories. Multiple users of electronic eyewear devices in an environment may simultaneously generate audio signals (e.g., voices/sounds) that are difficult to distinguish from one another. The electronic eyewear device tracks the location of moving remote electronic eyewear devices of other users, or an object of the other users, such as the remote user's face, to provide audio source separation using location of the sound sources. The simplified voice unmixing uses a microphone array of the electronic eyewear device and the known location of the remote user's electronic eyewear device with respect to the user's electronic eyewear device to facilitate audio source separation.

Eyewear with audio source separation using pose trackers

A method for AR-guided depth estimation is described. The method includes identifying a virtual object rendered in a first frame that is generated based on a first pose of an augmented reality (AR) device, determining a second pose of the AR device, the second pose following the first pose, identifying an augmentation area in the second frame based on the virtual object rendered in the first frame, and the second pose, determining depth information for the augmentation area in the second frame, and rendering the virtual object in the second frame based on the depth information.

Augmented reality guided depth estimation

An eXtended Reality (XR) display system includes a Light Emitting Diode (LED) display controller, and a Light Emitting Diode (LED) near-eye display element operatively coupled to the LED display driver. The LED near-eye display element includes one or more motors and an LED array operably connected to the one or more motors. During operation, the LED display driver receives video data including a rendered virtual object of an XR experience and generates LED array control signals based on the video data, the LED array control signals causing one or more LEDs of the LED array to be energized in a sequence. The LED display driver also generates synchronized motor control signals and simultaneously communicates the LED array control signals to the LED array and the synchronized motor control signals to the one or more motors causing the LED near-eye display element to display the rendered virtual object.

Growing community of inventors

Vienna, Austria

Georgios Evangelidis