Snap Inc.

Fedir Poliakov

Jianchao Yang

Samuel Edward Hare

Guohui Wang

Linjie Yang

Xuehan Xiong

Maksim Gusarov

Igor Kudriashov

William Brendel

Shah Tanmay Anilkumar

Lidiia Bogdanovych

Oleksandr Pyshchenko

Yurii Monastyrshyn

Xin Chen

Shubham Vij

Michael David Marr

Kai Chen

Jun Huang

Michael Ronald Cieslak

Farnaz Azmoodeh

Jiayao Yu

Omer Cansizoglu

Stanislav Minakov

Istvan Marton

Chintan Doshi

Justin Wilder

Cristian Zaloj

Yuechuan Li

The subject technology sends captured image data and metadata to a first set of computing processes for processing. The subject technology receives, asynchronously from a second set of computing processes, second metadata. The subject technology detects a signal indicating a command to stop image data capturing. The subject technology, in response to the detected signal, generating a composite AR content item based on the metadata and the captured image data. The subject technology provides the composite AR content item for rendering for display on an eyewear device.

Recording augmented reality content on an eyewear device

An audio control system can control interactions with an application or device using keywords spoken by a user of the device. The audio control system can use machine learning models (e.g., a neural network model) trained to recognize one or more keywords. Which machine learning model is activated can depend on the active location in the application or device. Responsive to detecting keywords, different actions are performed by the device, such as navigation to a pre-specified area of the application.

Device control using audio data

A machine learning system can generate an image mask (e.g., a pixel mask) comprising pixel assignments for pixels. The pixels can be assigned to classes, including, for example, face, clothes, body skin, or hair. The machine learning system can be implemented using a convolutional neural network that is configured to execute efficiently on computing devices having limited resources, such as mobile phones. The pixel mask can be used to more accurately display video effects interacting with a user or subject depicted in the image.

Generating an image mask using machine learning

A mobile device can generate real-time complex visual image effects using asynchronous processing pipeline. A first pipeline applies a complex image process, such as a neural network, to keyframes of a live image sequence. A second pipeline generates flow maps that describe feature transformations in the image sequence. The flow maps can be used to process non-keyframes on the fly. The processed keyframes and non-keyframes can be used to display a complex visual effect on the mobile device in real-time or near real-time.

Real-time tracking-compensated image effects

Systems, devices, media, and methods are presented for segmenting an image of a video stream with a client device, identifying an area of interest, generating a modified area of interest within one or more image, identifying a first set of pixels and a second set of pixels, and modifying a color value for the first set of pixels.

Image segmentation and modification of a video stream

Systems, devices, media, and methods are presented for determining performance metrics of client devices on a network using benchmark applications. Benchmark applications are distributed to client devices to produce performance metrics for the client devices. Performance metrics of the client devices received from the client devices are grouped to obtain performance metrics for different types of client devices.

Network device performance metrics determination

A machine learning system can generate an image mask (e.g., a pixel mask) comprising pixel assignments for pixels. The pixels can he assigned to classes, including, for example, face, clothes, body skin, or hair. The machine learning system can be implemented. using a convolutional neural network that is configured to execute efficiently on computing devices having limited resources, such as mobile phones. The pixel mask can be used to more accurately display video effects interacting with a user or subject depicted in the image.

Systems, devices, media, and methods are presented for presentation of modified objects within a video stream. The systems and methods receive a set of images within a video stream and identify at least a portion of a face in a first subset of images. The systems and methods determine face characteristics by analyzing the portion of the face in the first subset of images. The systems and methods apply a graphical representation of glasses to the face based on the face characteristics and cause presentation of a modified video stream including the portion of the face with the graphical representation of the glasses in a second subset of images of the set of images while receiving the video stream.

Modifying multiple objects within a video stream

Systems, devices, media, and methods are presented for segmenting an image of a video stream with a client device, binarizing an area of interest within one or more image, identifying an initial pupil location and an initial iris radius, and determining a final pupil location and a final iris radius. Some embodiments enable the client device to perform one or more operations within a user interface based on the image segmentation.

Image segmentation of a video stream

Systems, devices, media, and methods are presented for gaze-based control of device operations. One method includes receiving a video stream from an imaging device, the video stream depicting one or more eyes, determining a gaze direction for the one or more eyes depicted in the video stream, detecting a change in the gaze direction of the one or more eyes, and triggering an operation in a client device based on the change in the gaze direction.

Growing community of inventors

Marina Del Rey, CA, United States of America

Fedir Poliakov