Snap Inc.

Piotr Gurgul

Sharon Moll

Lucas Rangit Magasweran

Tomasz Zakrzewski

Dawei Zhang

Pawel Wawruch

Francis Patrick Sullivan

Andrei Rybin

AR-enabled wearable electronic devices such as smart glasses are adapted for use as an (Internet of Things) IoT device where the user can control a pointer and present augmented reality content with respect to a television screen, computer screen, or other IoT enabled device by selecting items by looking at them and making selections using gestures. Built-in six-degrees-of-freedom (6DoF) tracking capabilities are used to track the television display screen to facilitate cursor navigation and display of AR content. The tracking information is used to trigger an AR effect for display of augmented reality content relative to the physical television display screen (e.g., on or in the environment around the television display screen) on demand in response to gestures. The techniques are particularly useful for smart displays that offer AR-enhanced content that can be viewed in the displays of the AR-enabled wearable electronic devices.

AR glasses as IoT device for enhanced screen experience

An eXtended Reality (XR) system provides methodologies for capturing hand poses being made by a user in low-light environments. The XR system capture, using one or more visible light cameras, tracking video frame data of a hand pose of a user of the XR system. The XR system generates hand-tracking data based on the tracking video frame data. The hand-tracking data includes a skeletal model and a hand-tracking confidence level indicating a probability that the skeletal model matches the hand pose. The XR system compares the hand-tracking confidence level to a threshold confidence value, and based on determining the first hand-tracking confidence level is below the threshold confidence value, activates one or more wide-spectrum cameras to capture subsequent tracking video frame data of the hand pose. The XR system may also activate an IR light emitter to illuminate the hands of the user.

Hand-tracking with IR camera for XR systems

AR-enabled wearable electronic devices such as smart glasses are adapted for use as an (Internet of Things) IoT remote control device where the user can control a pointer on a television screen, computer screen, or other IoT enabled device to select items by looking at them and making selections using gestures. Built-in six-degrees-of-freedom (6DoF) tracking capabilities are used to move the pointer on the screen to facilitate navigation. The display screen is tracked in real-world coordinates to determine the point of intersection of the user's view with the screen using raycasting techniques. Hand and head gesture detection are used to allow the user to execute a variety of control actions by performing different gestures. The techniques are particularly useful for smart displays that offer AR-enhanced content that can be viewed in the displays of the AR-enabled wearable electronic devices.

AR glasses as IoT remote control

Systems, methods, and computer readable media for voice-controlled user interfaces (UIs) for augmented reality (AR) wearable devices are disclosed. Embodiments are disclosed that enable a user to interact with the AR wearable device without using physical user interface devices. An application has a non-voice-controlled UI mode and a voice-controlled UI mode. The user selects the mode of the UI. The application running on the AR wearable device displays UI elements on a display of the AR wearable device. The UI elements have types. Predetermined actions are associated with each of the UI element types. The predetermined actions are displayed with other information and used by the user to invoke the corresponding UI element.

Voice controlled UIs for AR wearable devices

Loading and unloading of ML models into an ML model cache or system memory of an electronic eyewear device is managed based on which applications are active or available and predicted activities. Sensor inputs are processed to detect whether the electronic eyewear device has moved or is predicted to move and new ML models are downloaded based on updated location information or observable visual information. Sensor inputs are also processed to determine whether the electronic eyewear device has changed state or resource availability and whether the ML model cache or system memory needs to be resized to accommodate new ML models for the changed conditions. If so, stored ML models are updated to reflect the new device state by unloading an ML model, receiving a new ML model based on the changed state or resource availability and a processing priority of the new ML model, or both.

Optimized ML model loading using predicted usage and device state

Systems, methods, and computer readable media for enhanced augmented reality (AR) reading are provided, where the methods include entering a reading mode, capturing an image of a section of reading materials such as a page of a book, identifying a code within the image, and identifying a code module corresponding to the code. The methods further include executing the code module, the code module providing an augmented reality object or virtual reality object related to an object depicted on the section or page, entering a sleep mode for a sleep duration based on an estimated reading time duration of a section or page, awaking after the sleep duration, and capturing a next image. The sleep duration is adjusted based on times when identifiers of the sections of the reading materials such as numbers are captured. The user may manually wake the AR reading device using an external user interface.

Enhanced reading with AR glasses

Systems, methods, and computer readable media that schedules requests for location data of a mobile device, where the methods include selecting a first positioning system based on a power requirement, a latency requirement, and an accuracy requirement, and determining whether a first condition is satisfied for querying the first positioning system. The method further comprises in response to a determination that the first condition is satisfied, querying the first positioning system for first position data. The method further comprises in response to a determination that the first condition is not satisfied, selecting a second positioning system based on the power requirement, the latency requirement, and the accuracy requirement, determining whether a second condition is satisfied for querying the second positioning system, and in response to a determination that the second condition is satisfied, querying the second positioning system for second position data.

Scheduling requests for location data

Systems, methods, and computer readable media for auto-recovery of an augmented reality (AR) wearable device are disclosed. A pass-through application is invoked as a background process and an application is invoked as a foreground process. The pass-through application includes an on-resume procedure that is called if the operating system or interpreter determines that the foreground process is unresponsive. The on-resume procedure restarts the application as the foreground process and may first reboot the AR wearable device. The pass-through application remains transparent to the user by not displaying output on the display of the AR wearable device. Additionally, an uncaught exception handler is registered with the operating system to be called in the event that an exception occurs that does not have a handler. The exception handler restarts the application as the foreground process and may first reboot the AR wearable device.

Auto-recovery for AR wearable devices

Apparatuses, computer readable medium, and methods for image capturing while circumnavigating objects using mobile devices are disclosed. Example methods include capturing an image, processing the image to identify an object within the image, determining a path around the object and a number of images to capture of the object, dividing the path by the number of images to determine a number of waypoints, and navigating the mobile device to the waypoints and capturing an image of the object at each waypoint of the waypoints. Examples include a person pointing at an object and the mobile device identifying the object based on the person pointing at the object. The mobile device determines a bounding box and a geometric center of the bounding box to determine the path to circumnavigate the object. The mobile device determines a height above a ground to assist in navigation.

Image capturing while circumnavigating objects using mobile devices

Systems, methods, and computer readable media for voice input for augmented reality (AR) wearable devices are disclosed. Embodiments are disclosed that enable a user to interact with the AR wearable device without using physical user interface devices. A keyword is used to indicate that the user is about to speak an action or command. The AR wearable device divides the processing of the audio data into a keyword module that is trained to recognize the keyword and a module to process the audio data after the keyword. In some embodiments, the AR wearable device transmits the audio data after the keyword to a host device to process. The AR wearable device maintains an application registry that associates actions with applications. Applications can be downloaded, and the application registry updated where the applications indicate actions to associate with the application.

Voice input for AR wearable devices

An architecture is provided for packaging visual overlay-based user interfaces (UIs) into mobile device applications to work as user interface extensions that allow certain flows and logic to be displayed on an eyewear device when connected to the mobile device application. The extension of the UIs of the mobile device applications to the display of the eyewear device allows for inexpensive experimentation with augmented reality (AR) UIs for eyewear devices and allows for reusing of business logic across mobile devices and associated eyewear devices. For example, a mobile device application for maps or navigation may be extended to show directions on an associated eyewear device once the destination is chosen in the navigation application on the mobile device. In this example, the business logic would still live in the navigation application on the mobile device but the user would see AR directions overlaid on a display of the eyewear device.

Extending user interfaces of mobile apps to AR eyewear

A head-worn device system includes one or more cameras, one or more display devices and one or more processors. The system also includes a memory storing instructions that, when executed by the one or more processors, configure the system to detect a gesture made by a user of the computing apparatus and generate gesture data identifying the gesture, select an application or selected action from a set of registered applications and actions based on the gesture data, and invoke the application or selected action.

Gesture-based application invocation

Systems, methods, and computer readable media for input modalities for an augmented reality (AR) wearable device are disclosed. The AR wearable device captures images using an image capturing device and processes the images to identify objects. The objects may be people, places, things, and so forth. The AR wearable device associates the objects with tags such as the name of the object or a function that can be provided by the selection of the object. The AR wearable device then matches the tags of the objects with tags associated with AR applications. The AR wearable device presents on a display of the AR wearable device indications of the AR applications with matching tags, which provides a user with the opportunity to invoke one of the AR applications. The AR wearable device recognizes a selection of an AR application in a number of different ways including gesture recognition and voice commands.

Input modalities for AR wearable devices

Systems, methods, and computer readable media that determine a location of a device using multi-source geolocation data, where the methods include accessing new location data from a location source of a plurality of location sources, where the new location data includes a new position and an accuracy of the new position, and determining a current position and an accuracy of the current position based on the new position, the accuracy of the new position, an previous current position, and an accuracy of the previous current position. The method further includes determining a change in location based on a difference between the current position and the previous current position. Some systems, methods, and computer readable media are directed to scheduling location requests to generate location data where the scheduling and the actual requests are made based on a number of conditions.

Determining location using multi-source geolocation data

A mixed-reality media content system may be configured to perform operations that include: causing display of image data at a client device, the image data comprising a depiction of an object that includes a graphical code at a position upon the object; detecting the graphical code at the position upon the depiction of the object based on the image data; accessing media content within a media repository based on the graphical code scanned by the client device; and causing display of a presentation of the media content at the position of the graphical code upon the depiction of the object at the client device.

Growing community of inventors

Hergiswil, Switzerland

Piotr Gurgul