Snap Inc.

Shin Hwun Kang

Ilteris Kaan Canberk

Dmytro Kucher

James Powderly

Dmytro Hovorov

Daniel Moreno

Karen Stolzenberg

Doug Mead

David Meisenholder

Bernhard Jung

Kristina Marrero

Lien Le Hong Tran

Terek Judi

Daria Skrypnyk

Kristian Bauer

Tiago Rafael Duarte

Yu Jiang Tham

Kyle Goodrich

Rajan Vaish

Kaveh Anvaripour

Sven Kratz

Brian Anthony Smith

Mathieu Emmanuel Vignau

Joseph Timothy Fortier

Viktoria Hwang

Artur Sydoran

Tianyi Sun

Ivan Fekete

Ihor Kuzin

Vernon James Carlos Manlapaz

Systems, methods, and computer readable media for an augmented reality (AR) shared screen space. Examples relate to a host augmented realty (AR) device sharing a screen and a relative location of the AR device to the screen with guest AR devices where the guest AR devices share a relative location of the guest AR devices to a copy of the screen displayed on the display of the guest AR devices and where the users of the AR devices may see each other's location with the use of avatars around the shared screen and add augmentations to the shared screen. The yaw, roll, and pitch of the head of the avatars tracks the movement of the head of the user of the AR wearable device.

Augmented reality shared screen space

Eyewear providing an interactive augmented reality experience to allow a user of an eyewear device to display a 3D overlay image on a viewed person. The user can select the overlay image from a list of images, such as costumes, stored in memory or generated by the user. The images can be sorted in memory based on common attributes. Registration points of the person are continuously aligned with registration points of the overlay as the person moves such that the user appears to be wearing the 3D costume during movement. By aligning the registration points, the costume adapts to different body types and heights. The coloring of the costume can change based on the environment, such as the lighting, or to contrast with colors viewed in a viewfinder.

Augmented reality eyewear with 3D costumes

Systems, devices, media, and methods are presented for an immersive augmented reality (AR) experience using an eyewear device. A portable eyewear device includes a processor, a memory, and a display projected onto at least one lens assembly. The memory has programming stored therein that, when executed by the processor, captures information depicting an environment surrounding the device and identifies a match between objects in that information and predetermined objects in previously obtained information for the same environment. When the position of the eyewear device reaches a preselected location with respect to the matched objects, a physical output is provided to produce the immersive experience. The physical output changes as the position of the eyewear device moves to maintain the immersive experience.

Augmented reality with eyewear triggered IoT

Systems and methods for controlling an Internet of Things (IoT) device through interaction with an augmented reality (AR) camera includes pairing an interactable augmented reality (AR) overlay with the IoT device. The interactable overlay includes control information for generating a control signal for controlling the IoT device when the interactable overlay is interacted with by the AR camera. The interactable overlay is presented on a display of the AR camera when the IoT device is in a field of view of the AR camera. An indication that the user has pointed the AR camera at the interactable overlay associated with the IoT device on the display of the AR camera is provided, and a control signal associated with the interactable overlay pointed at by the AR camera is provided to the at least one IoT device associated with the interactable overlay pointed at by the AR camera.

Interface for engaging IoT devices with AR camera

Systems, devices, media, and methods are presented for an immersive augmented reality (AR) experience using an eyewear device with spatial audio. The eyewear device has a processor, a memory, an image sensor, and speakers. The eyewear device captures image information for an environment surrounding the device, identifies a match between objects in the image information and predetermined objects in previously obtained information for the same environment. The eyewear device then identifies a target location within the environment, which may be associated with a physical or a virtual object. The eyewear device monitors its orientation with respect to the target location and presents audio signals to guide the user toward the target location.

Immersive augmented reality experiences using spatial audio

Eyewear presenting text corresponding to spoken words (e.g., in speech bubbles) and optionally translating from one language to another. In one example, an interactive augmented reality experience is provided between two users of eyewear devices to allow one user of an eyewear device to share a personal attribute of the user with a second user. The personal attribute can be speech spoken by a remote second user of eyewear converted to text. The converted text can be displayed on a display of eyewear of the first user proximate the viewed second user. The personal attribute may be displayed in a speech bubble proximate the second user, such as proximate the head or mouth of the second user. The language of the spoken speech can be recognized by the second user eyewear, and translated to a language that is understood by the first user. In another example, the spoken words of a remote person is captured by the eyewear of a user, the position of the remote person is identified, the spoken word are converted to text, and the text is displayed (e.g., in a speech bubble) on an AR display of the eyewear adjacent the remote person.

Augmented reality eyewear with speech bubbles and translation

Augmented reality guidance for guiding a user through an environment using an eyewear device. The eyewear device includes a display system and a position detection system. A user is guided though an environment by monitoring a current position of the eyewear device within the environment, identifying marker positions within a threshold of the current position, the marker positions defined with respect to the environment and associated with guidance markers, registering the marker positions, generating overlay image including the guidance markers, and presenting the overlay image on a display of the eyewear device.

Augmented reality guidance that generates guidance markers

Systems and methods are provided for performing operations on an augmented reality (AR) device using an external screen streaming system. The system establishes, by one or more processors of an AR device, a communication with an external client device. The system causes overlay of, by the AR device, a first AR object on a real-world environment being viewed using the AR device. The system receives, by the AR device, a first image from the external client device. The system, in response to receiving the first image from the external client device, overlays the first image on the first AR object by the AR device.

External screen streaming for an eyewear device

A head-worn device system includes one or more cameras, one or more display devices and one or more processors. The system also includes a memory storing instructions that, when executed by the one or more processors, configure the system to perform operations to initiate or join a joint visual computing session. The method may comprise receiving user input to initiate a joint session of a visual computing experience, monitoring for short-range data transmissions including data indicating the existence of a current session of the visual computing experience, and based on determining that a current session is in progress, providing a user input option to join the current session of the visual computing experience.

Shared augmented reality session creation

Devices, media, and methods are presented for an immersive augmented reality (AR) experience using an eyewear device with spatial audio. The eyewear device has a processor, a memory, an image sensor, and a speaker system. The eyewear device captures image information for an environment surrounding the device and identifies an object location within the same environment. The eyewear device then associates a virtual object with the identified object location. The eyewear device monitors the position of the device with respect to the virtual object and presents audio signals to alert the user that the identified object is in the environment.

Augmented reality spatial audio experience

An augmented reality (AR) content system is provided. The AR content system may analyze audio input obtained from a user to generate a search request. The AR content system may obtain search results in response to the search request and determine a layout by which to display the search results. The search results may be displayed in a user interface within an AR environment according to the layout. The AR content system may also analyze audio input to detect commands to perform with respect to content displayed in the user interface.

Generating user interfaces in augmented reality environments

Example systems, devices, media, and methods are described for evaluating movements and physical exercises in augmented reality using the display of an eyewear device. A motion evaluation application implements and controls the capturing of frames of motion data using an inertial measurement unit (IMU) on the eyewear device. The method includes presenting virtual targets on the display, localizing the current eyewear device location based on the captured motion data, and presenting virtual indicators on the display. The virtual targets represent goals or benchmarks for the user to achieve using body postures. The method includes detecting determining whether the eyewear device location represents an intersecting posture relative to the virtual targets, based on the IMU data. The virtual indicators display real-time feedback about user posture or performance relative to the virtual targets.

Virtual evaluation tools for augmented reality exercise experiences

Systems, devices, media, and methods are presented for producing an augmented reality (AR) experience for display on a smart eyewear device. The AR production system includes a marker registration utility for setting and storing markers, a localization utility for locating the eyewear device relative to a marker location and to the mapped environment, and a virtual object rendering utility to presenting one or more virtual objects having a desired size, shape, and orientation. A high-definition camera captures an input image of the environment. If the input image includes a marker, the system retrieves from memory a set of data including a first marker location expressed in terms relative to a marker coordinate system. The localization utility determines a local position of the eyewear device relative to the marker location. The virtual object rendering utility prepares one or more virtual objects for display based on the eyewear location, the head pose of the wearer, and the location of one or more physical object landmarks in the environment.

Marker-based guided AR experience

Systems and methods are described for selectively sharing audio and video streams amongst electronic eyewear devices. Each electronic eyewear device includes a camera arranged to capture a video stream in an environment of the wearer, a microphone arranged to capture an audio stream in the environment of the wearer, and a display. A processor of each electronic eyewear device executes instructions to establish an always-on session with other electronic eyewear devices and selectively shares an audio stream, a video stream, or both with other electronic eyewear devices in the session. Each electronic eyewear device also generates and receives annotations from other users in the session for display with the selectively shared video stream on the display of the electronic eyewear device that provided the selectively shared video stream. The annotation may include manipulation of an object in the shared video stream or overlay images registered with the shared video stream.

Hyper-connected and synchronized AR glasses

Example systems, devices, media, and methods are described for controlling virtual elements or graphical elements on a display in response to hand gestures detected by an eyewear device that is capturing frames of video data with its camera system. An image processing system detects a series of hand shapes in the video data and determines whether it matches a predefined series of hand gestures. Each predefined series of hand gestures is associated with an action. The system controls movement of the virtual element, relative to the display, in accordance with the associated action. In an example hand shape that includes a thumb sliding along an extended finger, the system establishes a finger scale along the extended finger, calibrates a graphical scale with the finger scale, and controls movement of an interactive graphical element, such as a slider, according to the current thumb position relative to the calibrated graphical scale.

Growing community of inventors

Playa Del Rey, CA, United States of America

Shin Hwun Kang