Snap Inc.

Kyle Goodrich

Andrew James McPhee

Daniel Moreno

Maxim Maximov Lazarov

Samuel Edward Hare

Tony Mathew

Wentao Shang

Dhritiman Sagar

Dylan Shane Eirinberg

Kaveh Anvaripour

Pedram Javidpour

Karen Stolzenberg

David Meisenholder

Mathieu Emmanuel Vignau

Piers George Cowburn

Isac Andreas Müller Sandvik

David Li

Joseph Timothy Fortier

Jeremy Baker Voss

Celia Nicole Mourkogiannis

Christie Marie Heikkinen

Vincent Sung

Sana Park

Alex Edelsburg

Tianyi Sun

Ilteris Kaan Canberk

Shin Hwun Kang

Aspects of the present disclosure involve a system comprising a computer-readable storage medium storing at least one program and a method for rendering virtual modifications to real-world environments depicted in image content. A reference surface is detected in a three-dimensional (3D) space captured within a camera feed produced by a camera of a computing device. An image mask is applied to the reference surface within the 3D space captured within the camera feed. A visual effect is applied to the image mask corresponding to the reference surface in the 3D space. The application of the visual effect to the image mask causes a modified surface to be rendered in presenting the camera feed on a display of the computing device.

Virtual surface modification

The subject technology receives frames of a source media content. The subject technology detects from the frames of the source media content, a first gesture indicating a cut point at a particular frame of the source media content, the cut point associated with a trimming operation to be performed on the source media content. The subject technology selects a starting frame and an ending frame from the frames based at least in part on the cut point at the particular frame. The subject technology performs the trimming operation based on the starting frame and the ending frame. The subject technology generates a second media content using the third set of frames. The subject technology provides for display at least a portion of the third set of frames of the second media content.

Auto trimming for augmented reality content in messaging systems

The subject technology detects a first gesture corresponding to an open trigger finger gesture. The subject technology detects a location and a position of a representation of a finger from the open trigger finger gesture. The subject technology generates a first virtual object based at least in part on the location and the position of the representation of the finger. The subject technology detects a first collision event. The subject technology detects a second gesture corresponding to a closed trigger finger gesture. The subject technology selects the second virtual object. The subject technology renders the first virtual object as attached to the second virtual object in response to the selecting. The subject technology provides for display the rendered first virtual object as attached to the second virtual object within a first scene.

Trigger gesture for selection of augmented reality content in messaging systems

The subject technology receives a set of frames. The subject technology detect a first gesture correspond to an open trigger finger gesture. The subject technology receives a second set of frames. The subject technology detects from the second set of frames, a second gesture correspond to a closed trigger finger gesture. The subject technology detects a location and a position of a representation of a finger from the closed trigger finger gesture. The subject technology generates a first virtual object based at least in part on the location and the position of the representation of the finger. The subject technology renders a movement of the first virtual object along a vector away from the location and the position of the representation of the finger within a first scene. The subject technology provides for display the rendered movement of the first virtual object along the vector within the first scene.

Shooting interaction using augmented reality content in a messaging system

The subject technology applies a three-dimensional (3D) effect to image data and depth data based at least in part on an augmented reality content generator. The subject technology generates a segmentation mask based at least on the image data. The subject technology performs background inpainting and blurring of the image data using at least the segmentation mask to generate background inpainted image data. The subject technology generates a packed depth map based at least in part on the a depth map of the depth data. The subject technology generates, using the processor, a message including information related to the applied 3D effect, the image data, and the depth data.

Generating 3D data in a messaging system

Aspects of the present disclosure involve a system comprising a computer-readable storage medium storing at least one program and method for performing operations comprising: receiving, by a messaging application, a video feed from a camera of a user device that depicts a face; receiving a request to add a 3D caption to the video feed; identifying a graphical element that is associated with context of the 3D caption; and displaying the 3D caption and the identified graphical element in the video feed at a position in 3D space of the video feed proximate to the face depicted in the video feed.

3D captions with semantic graphical elements

Methods and systems are disclosed for performing operations for generating a custom sticker. The operations include receiving, by a messaging application implemented on a first device, an image. The operations include receiving, by the messaging application, input that selects a region of the image. The operations include generating, by the messaging application, a graphical element comprising the region of the image selected by the input. The operations include applying, by the messaging application, one or more visual effects to the graphical element. The operations include causing the graphical element to be animated according to a given animation pattern selected from a plurality of animation patterns to generate an animated custom graphic. The operations include sending the animated custom graphic with the one or more visual effects from the first device to a second device.

Animated custom sticker creation

The subject technology receives first image data captured by a camera of an eyewear device. The subject technology detects, using a machine learning model, a representation of a display screen in the first image data. The subject technology selects at least a portion of the representation of the display screen. The subject technology adjusts a visual appearance of the portion of the representation of the display screen. The subject technology causes display of the adjusted visual appearance using a display system of the eyewear device.

Detection and obfuscation of display screens in augmented reality content

The subject technology determines a gaze direction in a field of view of a user using an eyewear device. The subject technology generates an anchor point in the field of view based at least in part on the determined gaze direction. The subject technology identifies a surface corresponding to a ground plane in the field of view. The subject technology determines a distance from the identified surface to the anchor point. The subject technology generates AR content based at least in part on the determined distance. The subject technology renders the generated AR content in the field of view for display by the eyewear device.

Determining gaze direction to generate augmented reality content

A messaging system, which hosts a backend service for an associated messaging client application, includes a voice notes system that addresses the technical problem of serving an audio message to the recipient in a manner that permits the recipient to consume the message in a text format as a transcription. A chat user interface (UI) provided with the voice notes system permits a user to play an audio message or request generation of the transcription of the audio message on-demand to prevent unnecessary cluttering of the UI real estate.

Voice notes with transcription

Methods and systems are disclosed for generating AR experiences on a messaging platform. The methods and systems perform operations including: detecting a real-world object depicted in a first image captured by a first camera of a client device, the client device comprising a second camera; extracting one or more textures from the real-world object depicted in the first image; selecting a target object depicted in a second image captured by the second camera, the second image being captured by the second camera simultaneously with the first image captured by the first camera; generating an augmented reality (AR) element comprising the target object modified based on the one or more textures extracted from the real-world object depicted in the first image; and causing display of the AR element within the second image.

Augmented reality experiences with dual cameras

The subject technology receives, at a client device, a selection of a selectable graphical item from a plurality of selectable graphical items, the selectable graphical item comprising an augmented reality content generator including a 3D effect. The subject technology applies, to image data and depth data, the 3D effect based at least in part on the augmented reality content generator, the applying the 3D effect. The subject technology generates a depth map using at least the depth data, generates a segmentation mask based at least on the image data, and performs background inpainting and blurring of the image data using at least the segmentation mask to generate background inpainted image data. The subject technology generates a 3D message based at least in part on the applied 3D effect.

Effects for 3D data in a messaging system

Systems and methods are provided for performing operations comprising: displaying one or more virtual objects on an eyewear device; detecting, by a camera of the eyewear device, a hand within view of the eyewear device; determining that a first finger of the hand is oriented along a same direction as the one or more virtual objects that are displayed; detecting movement of a second finger of the hand relative to the first finger; and controlling the display of the one or more virtual objects in response to detecting movement of the second finger to the first finger.

Gesture control on an eyewear device

Systems and methods are provided for capturing by a camera of a user device, a first image depicting a first environment of the user device; overlaying a first virtual object on a portion of the first image depicting the first environment; modifying a surface of the first virtual object using content captured by the user device; storing a second virtual object comprising the first virtual object with the modified surface; and generating for display the second virtual object on a portion of a second image depicting a second environment.

3D object camera customization system

Aspects of the present disclosure involve a system comprising a computer-readable storage medium storing at least one program and method for performing operations comprising: receiving, by a messaging application, an image from a camera of a user device; receiving input that selects a user-customizable effects option for activating a user-customizable effects mode; in response to receiving the input, displaying an array of a plurality of effect options together with the image proximate to the user-customizable effects option; and applying a first effect associated with a first effect option of the plurality of effect options to the image.

Growing community of inventors

Venice, CA, United States of America

Kyle Goodrich