Snap Inc.

Sharon Moll

Piotr Gurgul

Dawei Zhang

Tomasz Zakrzewski

Pawel Wawruch

Francis Patrick Sullivan

Andrei Rybin

Neken Aritia Symphonie Razafindrabe

AR-enabled wearable electronic devices such as smart glasses are adapted for use as an (Internet of Things) IoT device where the user can control a pointer and present augmented reality content with respect to a television screen, computer screen, or other IoT enabled device by selecting items by looking at them and making selections using gestures. Built-in six-degrees-of-freedom (6DoF) tracking capabilities are used to track the television display screen to facilitate cursor navigation and display of AR content. The tracking information is used to trigger an AR effect for display of augmented reality content relative to the physical television display screen (e.g., on or in the environment around the television display screen) on demand in response to gestures. The techniques are particularly useful for smart displays that offer AR-enhanced content that can be viewed in the displays of the AR-enabled wearable electronic devices.

AR glasses as IoT device for enhanced screen experience

An eXtended Reality (XR) system provides methodologies for capturing hand poses being made by a user in low-light environments. The XR system capture, using one or more visible light cameras, tracking video frame data of a hand pose of a user of the XR system. The XR system generates hand-tracking data based on the tracking video frame data. The hand-tracking data includes a skeletal model and a hand-tracking confidence level indicating a probability that the skeletal model matches the hand pose. The XR system compares the hand-tracking confidence level to a threshold confidence value, and based on determining the first hand-tracking confidence level is below the threshold confidence value, activates one or more wide-spectrum cameras to capture subsequent tracking video frame data of the hand pose. The XR system may also activate an IR light emitter to illuminate the hands of the user.

Hand-tracking with IR camera for XR systems

AR-enabled wearable electronic devices such as smart glasses are adapted for use as an (Internet of Things) IoT remote control device where the user can control a pointer on a television screen, computer screen, or other IoT enabled device to select items by looking at them and making selections using gestures. Built-in six-degrees-of-freedom (6DoF) tracking capabilities are used to move the pointer on the screen to facilitate navigation. The display screen is tracked in real-world coordinates to determine the point of intersection of the user's view with the screen using raycasting techniques. Hand and head gesture detection are used to allow the user to execute a variety of control actions by performing different gestures. The techniques are particularly useful for smart displays that offer AR-enhanced content that can be viewed in the displays of the AR-enabled wearable electronic devices.

AR glasses as IoT remote control

Systems, methods, and computer readable media for voice-controlled user interfaces (UIs) for augmented reality (AR) wearable devices are disclosed. Embodiments are disclosed that enable a user to interact with the AR wearable device without using physical user interface devices. An application has a non-voice-controlled UI mode and a voice-controlled UI mode. The user selects the mode of the UI. The application running on the AR wearable device displays UI elements on a display of the AR wearable device. The UI elements have types. Predetermined actions are associated with each of the UI element types. The predetermined actions are displayed with other information and used by the user to invoke the corresponding UI element.

Voice controlled UIs for AR wearable devices

A gesture-based text entry user interface for an Augmented Reality (AR) device is provided. The AR system detects a start text entry gesture made by a user of the AR system, generates a virtual keyboard user interface including a virtual keyboard having a plurality of virtual keys, and provides to the user the virtual keyboard user interface. The AR system determines using the one or more cameras, the user's selection of one or more selected virtual keys of the plurality of virtual keys and generates entered text data based on the one or more selected virtual keys. The AR system provides the entered text data to the user using a display of the AR system.

AR-based virtual keyboard

A gesture-based text entry user interface for an Augmented Reality (AR) system is provided. The AR system detects a start text entry gesture made by a user of the AR system, generates a virtual keyboard user interface including a virtual keyboard having a plurality of virtual keys, and provides to the user the virtual keyboard user interface. The AR system detects a hold of an enter text gesture made by the user. While the user holds the enter text gesture, the AR system collects continuous motion gesture data of a continuous motion as the user makes the continuous motion through the virtual keys of the virtual keyboard. The AR system detects a release of the enter text gesture by the user and generates entered text data based on the continuous motion gesture data.

Gesture-based keyboard text entry

An Augmented Reality (AR) graphics system is provided. The AR graphics system may coordinate the display of augmented reality graphics created by multiple users located in an environment. The AR graphics system may determine an alignment object located in the environment that is designated as a common origin of a real-world coordinate system that is used to determine where to display AR graphics within the environment. Additionally, a prioritization scheme is implemented to resolve conflicts between overlapping input provided by different users in order to generate a single version of AR graphics.

Generating user interfaces displaying augmented reality graphics

An Augmented Reality (AR) system is provided. The AR system generates a text scene object including virtual text objects based on a text and detect a text selection gesture made by a user of the AR system. The AR system generates a selection line based on a landmark of the user's hand. The AR system detects a confirmation gesture made by the user and sets a text selection start point at an intersection of the selection line with a start virtual text object of the one or more virtual text objects. The AR system detects a subsequent confirmation gesture made by the user, sets a text selection end point, and selects selected text from the text based on the text selection start point and the text selection end point.

Hand-tracked text selection and modification

Apparatuses, computer readable medium, and methods for image capturing while circumnavigating objects using mobile devices are disclosed. Example methods include capturing an image, processing the image to identify an object within the image, determining a path around the object and a number of images to capture of the object, dividing the path by the number of images to determine a number of waypoints, and navigating the mobile device to the waypoints and capturing an image of the object at each waypoint of the waypoints. Examples include a person pointing at an object and the mobile device identifying the object based on the person pointing at the object. The mobile device determines a bounding box and a geometric center of the bounding box to determine the path to circumnavigate the object. The mobile device determines a height above a ground to assist in navigation.

Image capturing while circumnavigating objects using mobile devices

Systems, methods, and computer readable media for voice input for augmented reality (AR) wearable devices are disclosed. Embodiments are disclosed that enable a user to interact with the AR wearable device without using physical user interface devices. A keyword is used to indicate that the user is about to speak an action or command. The AR wearable device divides the processing of the audio data into a keyword module that is trained to recognize the keyword and a module to process the audio data after the keyword. In some embodiments, the AR wearable device transmits the audio data after the keyword to a host device to process. The AR wearable device maintains an application registry that associates actions with applications. Applications can be downloaded, and the application registry updated where the applications indicate actions to associate with the application.

Voice input for AR wearable devices

An Augmented Reality (AR) graphics system is provided. The AR graphics system may detect an object in a real-world scene that corresponds to an AR graphics display surface. The AR graphics system may generate AR graphics that are displayed as overlays of the AR graphics display surface. The AR graphics system may track the motion of a graphics input tool with respect to the AR graphics display surface to generate AR graphics based on the motion of the graphics input tool. The AR graphics may be comprised of a number of markings generated based on the motion of the graphics input tool.

A resource optimized kiosk mode that improves the mobile experience for creators and users of mobile devices such as an augmented reality (AR)-enabled wearable eyewear device. An eyewear device enters a kiosk mode by receiving a kiosk mode request for an application and, in response to the request, determining which services and application programming interfaces (APIs) are required to execute the selected application. An identification of the determined services and APIs required to execute the selected application are stored and the eyewear device is rebooted. After reboot, the selected application is started, and only the identified services and APIs are enabled. To determine which services and APIs are required to execute the selected application, metadata may be associated with the selected application specifying the services and/or APIs that the selected application requires to use when in operation.

Mobile device resource optimized kiosk mode

An architecture is provided for packaging visual overlay-based user interfaces (UIs) into mobile device applications to work as user interface extensions that allow certain flows and logic to be displayed on an eyewear device when connected to the mobile device application. The extension of the UIs of the mobile device applications to the display of the eyewear device allows for inexpensive experimentation with augmented reality (AR) UIs for eyewear devices and allows for reusing of business logic across mobile devices and associated eyewear devices. For example, a mobile device application for maps or navigation may be extended to show directions on an associated eyewear device once the destination is chosen in the navigation application on the mobile device. In this example, the business logic would still live in the navigation application on the mobile device but the user would see AR directions overlaid on a display of the eyewear device.

Extending user interfaces of mobile apps to AR eyewear

A head-worn device system includes one or more cameras, one or more display devices and one or more processors. The system also includes a memory storing instructions that, when executed by the one or more processors, configure the system to detect a gesture made by a user of the computing apparatus and generate gesture data identifying the gesture, select an application or selected action from a set of registered applications and actions based on the gesture data, and invoke the application or selected action.

Gesture-based application invocation

Systems, methods, and computer readable media for input modalities for an augmented reality (AR) wearable device are disclosed. The AR wearable device captures images using an image capturing device and processes the images to identify objects. The objects may be people, places, things, and so forth. The AR wearable device associates the objects with tags such as the name of the object or a function that can be provided by the selection of the object. The AR wearable device then matches the tags of the objects with tags associated with AR applications. The AR wearable device presents on a display of the AR wearable device indications of the AR applications with matching tags, which provides a user with the opportunity to invoke one of the AR applications. The AR wearable device recognizes a selection of an AR application in a number of different ways including gesture recognition and voice commands.

Growing community of inventors

Lachen, Switzerland

Sharon Moll