Snap Inc.

Matan Zohar

Itamar Berger

Gal Dudovitch

Avihay Assouline

Gal Sasson

Ma'ayan Mishin Shuvi

Brian Fulkerson

Yanli Zhao

Qi Pan

Isac Andreas Müller Sandvik

Sagi Katz

Ilya Levin

Jean Luo

Piers George Cowburn

David Li

Riza Alp Guler

Haoyang Wang

Peleg Harel

Maria Jose Garcia Sopo

Georgios Papandreou

Amar Boghani

Omri Berg

Nir Malbin

Artem Bondich

Antonios Kakolyris

Frank Lu

Aspects of the present disclosure involve a system for automatically zooming in on objects in an image. The system accesses a video comprising a frame depicting a person and detects a fashion item being worn by the person in the frame of the video. The system, in response to detecting the fashion item being worn by the person, extracts, from the frame, a portion comprising a depiction of the fashion item being worn by the person. The system generates a focus image of the fashion item using the portion extracted from the frame and presents the focus image of the fashion item together with the video comprising the frame depicting the person.

Adaptive zoom try-on experience

Methods and systems are disclosed for performing operations for applying augmented reality elements to a fashion item. The operations include receiving an image that includes a depiction of a person wearing a fashion item. The operations include generating a segmentation of the fashion item worn by the person depicted in the image. The operations include extracting a portion of the image corresponding to the segmentation of the fashion item; estimating an angle of each pixel in the portion of the image relative to a camera used to capture the image. The operations include applying one or more augmented reality elements to the fashion item in the image based on the estimated angle of each pixel in the portion of the image relative to the camera used to capture the image.

Body normal network light and rendering control

A virtual wave creation system comprises an eyewear device that includes a frame, a temple connected to a lateral side of the frame, and a depth-capturing camera. Execution of programming by a processor configures the virtual wave creation system to generate, for each of multiple initial depth images, a respective wave image by applying a transformation function that is responsive to a selected input parameter to the initial three-dimensional coordinates. The virtual wave creation system creates a warped wave video including a sequence of the generated warped wave images. The virtual wave creation system presents, via an image display, the warped wave video.

Input parameter based image waves

Aspects of the present disclosure involve a system for synthetically generating a view of an object to present one or more AR fashion items in an image. The system accesses a first image depicting a first portion of a body of a person from a first camera view. The system detects a fashion item in the first image that is being worn by the person in the first image, the fashion item being depicted in the first image from the first camera view. The system generates an augmented reality (AR) item that visually resembles the fashion item from the second camera view and modifies the second image with the AR item to present a synthetic view of the second portion of the body of the person wearing the AR item corresponding to the fashion item.

Synthetic view for try-on experience

Methods and systems are disclosed for performing operations comprising: receiving a video that includes a depiction of a person wearing a fashion item; generating a three-dimensional (3D) model of the person depicted in the video; generating a segmentation of the fashion item worn by the person depicted in the video; outlining a portion of the 3D model based on the segmentation of the fashion item; tracking movement of the portion of the 3D model in the video; and in response to tracking the movement of the portion of the 3D model in the video, modifying a display position of one or more augmented reality elements on the fashion item in the video.

3D upper garment tracking

Methods and systems are disclosed for performing operations comprising: receiving an image that includes a depiction of a face of a user; generating a plurality of landmarks of the face based on the received image; removing a set of interfering landmarks from the plurality of landmarks resulting in a remaining set of landmarks of the plurality of landmarks; obtaining a depth map for the face of the user; and computing a real-world scale of the face of the user based on the depth map and the remaining set of landmarks.

True size eyewear in real time

Methods and systems are disclosed for performing operations comprising: receiving an image that includes a depiction of a person wearing a fashion item; generating a segmentation of the fashion item by the person depicted in the image; receiving voice input associated with the person depicted in the image; in response to receiving the voice input, generating one or more augmented reality elements representing the voice input; and applying the one or more augmented reality elements to the fashion item worn by the person based on the segmentation of the fashion item worn by the person.

Controlling interactive fashion based on voice

Methods and systems are disclosed for performing operations comprising: receiving a video that includes a depiction of a real-world object; generating a three-dimensional (3D) body mesh associated with the real-world object that tracks movement of the real-world object across frames of the video; obtaining an external mesh associated with an augmented reality element; automatically establishing a correspondence between the 3D body mesh associated with the real-world object and the external mesh; deforming the external mesh based on movement of the real-world object and the established correspondence with the 3D body mesh; and modifying the video to include a display of the augmented reality element based on the deformed external mesh.

Deforming custom mesh based on body mesh

Methods and systems are disclosed for performing operations comprising: receiving an image that includes a depiction of a person wearing a fashion item; generating a segmentation of the fashion item worn by the person depicted in the image; identifying a facial expression of the user depicted in the image; and in response to identifying the facial expression, applying one or more augmented reality elements to the fashion item worn by the person based on the segmentation of the fashion item worn by the person.

Controlling interactive fashion based on facial expressions

Aspects of the present disclosure involve a system for providing an AR try-on experience for a friend. The system accesses a plurality of images that depict one or more persons. The system receives input that identifies a given person of the one or more persons who is depicted in an individual image of the plurality of images. The system extracts features of the given person depicted in the individual image. The system applies a machine learning model to the extracted features of the given person to generate an avatar that resembles the given person. The system applies one or more augmented reality (AR) fashion items to the avatar to generate an image that resembles the given person wearing the one or more AR fashion items.

Augmented reality try-on experience for friend

Aspects of the present disclosure involve a system for providing virtual experiences. The system accesses, by a messaging application, an image depicting a person. The system generates, by the messaging application, a three-dimensional (3D) avatar based on the person depicted in the image. The system receives input that selects a pose for the 3D avatar and one or more fashion items to be worn by the 3D avatar and places, by the messaging application, the 3D avatar in the selected pose and wearing the one or more fashion items in an augmented reality (AR) experience.

Applying animated 3D avatar in AR experiences

Aspects of the present disclosure involve a system comprising a computer-readable storage medium storing at least one program, and a method for performing operations comprising receiving an image that depicts a person, identifying a set of skeletal joints of the person and identifying a pose of the person depicted in the image based on positioning of the set of skeletal joints. The operations also include receiving speech input comprising a request to perform an AR operation and an ambiguous intent, discerning the ambiguous intent of the speech input based on the pose of the person depicted in the image and in response to receiving the speech input, performing the AR operation based on discerning the ambiguous intent of the speech input based on the pose of the person depicted in the image.

Inferring intent from pose and speech input

Aspects of the present disclosure involve a system comprising a computer-readable storage medium storing at least one program, and a method for performing operations comprising: receiving a video that depicts a person; tracking three-dimensional (3D) movement of the person within the video using a 3D reference point; computing a 3D position for placement of an augmented reality item relative to the 3D reference point; causing to be displayed the augmented reality item within the video at the 3D position; and updating the 3D position of the augmented reality item in the video as the 3D reference point changes based on the 3D movement of the person.

Side-by-side character animation from realtime 3D body motion capture

Aspects of the present disclosure involve a system comprising a computer-readable storage medium storing at least one program, and a method for performing operations comprising: receiving, by a client device associated with a first user, a communication from a second user; retrieving, from the communication, a movement vector representing three-dimensional (3D) movement of a set of skeletal joints of the second user; receiving, by the client device associated with the first user, input that selects a 3D avatar; and animating, based on the movement vector, the 3D avatar to mimic the 3D movement of the set of skeletal joints of the second user.

Body animation sharing and remixing

Aspects of the present disclosure involve a system comprising a computer-readable storage medium storing at least one program, and a method for performing operations comprising: receiving a video that depicts a person. The operations further include identifying a set of skeletal joints of the person. The operations further include identifying a pose of the person depicted in the video based on positioning of the set of skeletal joints (or detecting a hand pose, detecting a mirror frame, or detecting a mobile device). The operations further include determining, based on the pose of the person (or detecting a hand pose, detecting a mirror frame, or detecting a mobile device), that the video comprises a mirror reflection of the person. The operations further include, in response to determining that the video comprises the mirror reflection of the person, causing display of a 3D virtual object in the video.

Growing community of inventors

Rishon Lezion, Israel

Matan Zohar