Snap Inc.

Lei Zhang

Yu Jiang Tham

Rajan Vaish

Andrés Monroy-Hernández

Ava Robinson

Youjean Cho

Daekun Kim

Sven Kratz

Fannie Liu

Tianying Chen

Timothy Chong

Olivia Seow

Andrew Grosvenor Cooper

Yamill Antonio Vallecillo

Joseph Burfitt

Sivakumar Loganathan

Gene Michael Cahill, Jr

Augmented reality (AR) systems, devices, media, and methods are described for capturing and presenting effort put into generating a handcrafted AR experience. AR object generation data is captured during the generation of a handcrafted AR object. The AR object generation data is then processed to generate proof of effort data for inclusion with the handcrafted AR object. Examples of proof of effort include a time lapse view of the steps taken during generation of the AR object and statistics such as total time spent, number of images or songs considered for selection, number of actions implemented, etc.

Handcrafted augmented reality effort evidence

Augmented reality (AR) systems, devices, media, and methods are described for generating AR experiences including interactions with virtual or physical prop objects. The AR experiences are generated by capturing images of a scene with a camera system, identifying an object receiving surface and corresponding surface coordinates within the scene, identifying an AR primary object and a prop object (physical or virtual), establishing a logical connection between the AR primary object and the prop object, generating AR overlays including actions associated with the AR primary object responsive to commands received via a user input system that position the AR primary object adjacent the object receiving surface responsive to the primary object coordinates and the surface coordinates within the scene and that position the AR primary object and the prop object with respect to one another in accordance with the logical connection, and presenting the generated AR overlays with a display system.

Augmented reality prop interactions

A method for detecting full-body gestures by a mobile device includes a host mobile device detecting the tracked body of a co-located participant in a multi-party session. When the participant's tracked body provides a full-body gesture, the host's mobile device recognizes that there is a tracked body providing a full-body gesture. The host mobile device iterates through the list of participants in the multi-party session and finds the closest participant mobile device with respect to the screen-space position of the head of the gesturing participant. The host mobile device then obtains the user ID of the closest participant mobile device and broadcasts the recognized full-body gesture event to all co-located participants in the multi-party session, along with the obtained user ID. Each participant's mobile device may then handle the gesture event as appropriate for the multi-party session. For example, a character or costume may be assigned to a gesturing participant.

Co-located full-body gestures

A panoramic system can receive content (e.g., live video) recorded in a landscape orientation and use a portrait dynamic crop window to display a portrait area of the received content. The panoramic system can further display the content in a landscape dynamic crop window that has a moveable user interface control that can pan the portrait crop window as the content plays in one or both of the dynamic crop windows.

Simulated interactive panoramas

Augmented reality (AR) systems, devices, media, and methods are described for creating a handcrafted AR experience. The handcrafted AR experiences are created by capturing images of a scene, identifying an object receiving surface and corresponding surface coordinates, identifying a customizable AR primary object associated with at least one set of primary object coordinates, generating AR overlays including the customizable AR primary object for positioning adjacent the object receiving surface, presenting the AR overlays, receiving customization commands, generating handcrafted AR overlays including customizations associated with the customizable AR primary object responsive to the customization commands, presenting the handcrafted AR overlays, recording the handcrafted AR overlays, creating a handcrafted AR file including the recorded overlays, and transmitting the handcrafted AR file.

Handcrafted augmented reality experiences

Multi-player co-located AR experiences are augmented by assigning characters and costumes to respective participants (a.k.a. 'users' of AR-enabled mobile devices) in multi-player AR sessions for storytelling, play acting, and the like. Body tracking technology and augmented reality (AR) software are used to decorate the bodies of the co-located participants with virtual costumes within the context of the multi-player co-located AR experiences. Tracked bodies are distinguished to determine which body belongs to which user and hence which virtual costume belongs to which tracked body so that corresponding costumes may be assigned for display in augmented reality. A host-guest mechanism is used for networked assignment of characters and corresponding costumes in the co-located multi-player AR session. Body tracking technology is used to move the costume with the body as movement of the assigned body is detected.

Character and costume assignment for co-located users

Described are authoring tools for creating interactive AR experiences. The story-authoring application enables a user with little or no programming skills to create an interactive story that includes recording voice commands for advancing to the next scene, inserting and manipulating virtual objects in a mixed-reality environment, and recording a variety of interactions with connected IoT devices. The story creation interface is presented on the display as a virtual object in an AR environment.

Authoring tools for creating interactive AR experiences

Augmented reality (AR) systems, devices, media, and methods are described for sending and receiving AR objects (e.g., customized AR objects) based on/responsive to interactions with the physical world. AR virtual delivery route overlays are generated responsive to selected virtual delivery routes and include the AR object and a delivery mode (air, tunnel, etc.) corresponding to the virtual delivery route. Physical world actions associated with the delivery mode (blowing adjacent an AR device or scratching a surface) result in sending a communication corresponding to the AR object for delivery to a receiver and generating AR sending overlays including the AR object moving in accordance with the delivery mode.

Physical action-based augmented reality communication exchanges

A virtual interface application presented in augmented reality (AR) is described for controlling Internet of Things (IoT) products. The virtual interface application enables a user of a portable electronic device to activate a virtual control interface overlay on a display, receive a selection from the user using her hands or feet, and send a control signal to a nearby IoT product which executes an action in accordance with the selection. The virtual control interface is presented on the display as a virtual object in an AR environment. The virtual interface application includes a foot tracking tool for detecting an intersection between the foot location (in the physical environment) and the virtual surface position (in the AR environment). When an intersection is detected, the virtual interface application sends a control signal with instructions to the IoT product.

Virtual interfaces for controlling IoT devices

Described are virtual AR interfaces for generating a virtual rotational interface for the purpose of controlling connected IoT devices using the inertial measurement unit (IMU) of a portable electronic device. The IMU control application enables a user of a portable electronic device to activate a virtual rotational interface overlay on a display and adjust a feature of a connected IoT product by rotating a portable electronic device. The device IMU moves a slider on the virtual rotational interface. The IMU control application sends a control signal to the IoT product which executes an action in accordance with the slider position. The virtual rotational interface is presented on the display as a virtual object in an AR environment. The IMU control application detects the device orientation (in the physical environment) and in response presents a corresponding slider element on the virtual rotational interface (in the AR environment).

Growing community of inventors

Mountain View, CA, United States of America

Lei Zhang