L'oreal

Edmund Phung

Parham Aarabi

Irina Kezele

Alex Levinshtein

Eric Elmoznino

Ruowei Jiang

Cheng Chang

Zhi Yu

Tian Xing Li

Wenzhangzhi Guo

Brendan Duke

He Ma

Abdalla Ahmed

Methods, apparatus and techniques shape localized objects using a shape model for rendering a reshaped object with an effect such as for a virtual try on (VTO) experience. A VTO can simulate finger/toe nail effects of a nail product or service to nail objects. An example system comprises a nail localization engine including computational circuitry to localize one or more nail objects in an input image of a hand or foot via one or more deep neural networks; a nail shaping engine including computational circuitry for reshaping, via a trained shape model, the one or more nail object as localized by the nail localization engine; and a rendering component including computational circuitry to render an output image simulating a nail product or nail service applied to the one or more nail objects in accordance with the reshaping by the nail shaping engine to provide a virtual try on experience.

Method and system for effect contouring in an augmented reality experience

There are provided systems and methods for facial landmark detection using a convolutional neural network (CNN). The CNN comprises a first stage and a second stage where the first stage produces initial heat maps for the landmarks and initial respective locations for the landmarks. The second stage processes the heat maps and performs Region of Interest-based pooling while preserving feature alignment to produce cropped features. Finally, the second stage predicts from the cropped features a respective refinement location offset to each respective initial location. Combining each respective initial location with its respective refinement location offset provides a respective final coordinate (x,y) for each respective landmark in the image. Two-stage localization design helps to achieve fine-level alignment while remaining computationally efficient. The resulting architecture is both small enough in size and inference time to be suitable for real-time web applications such as product simulation and virtual reality.

Convolution neural network based landmark tracker

A system and method implement deep learning on a mobile device to provide a convolutional neural network (CNN) for real time processing of video, for example, to color hair. Images are processed using the CNN to define a respective hair matte of hair pixels. The respective object mattes may be used to determine which pixels to adjust when adjusting pixel values such as to change color, lighting, texture, etc. The CNN may comprise a (pre-trained) network for image classification adapted to produce the segmentation mask. The CNN may be trained for image segmentation (e.g. using coarse segmentation data) to minimize a mask-image gradient consistency loss. The CNN may further use skip connections between corresponding layers of an encoder stage and a decoder stage where shallower layers in the encoder, which contain high-res but weak features are combined with low resolution but powerful features from deeper decoder layers.

System and method for image processing using deep neural networks

This document relates to hybrid eye center localization using machine learning, namely cascaded regression and hand-crafted model fitting to improve a computer. There are proposed systems and methods of eye center (iris) detection using a cascade regressor (cascade of regression forests) as well as systems and methods for training a cascaded regressor. For detection, the eyes are detected using a facial feature alignment method. The robustness of localization is improved by using both advanced features and powerful regression machinery. Localization is made more accurate by adding a robust circle fitting post-processing step. Finally, using a simple hand-crafted method for eye center localization, there is provided a method to train the cascaded regressor without the need for manually annotated training data. Evaluation of the approach shows that it achieves state-of-the-art performance.

System and method using machine learning for iris tracking, measurement, and simulation

Systems and methods relate to a network model to apply an effect to an image such as an augmented reality effect (e.g. makeup, hair, nail, etc.). The network model uses a conditional cycle-consistent generative image-to-image translation model to translate images from a first domain space where the effect is not applied and to a second continuous domain space where the effect is applied. In order to render arbitrary effects (e.g. lipsticks) not seen at training time, the effect's space is represented as a continuous domain (e.g. a conditional variable vector) learned by encoding simple swatch images of the effect, such as are available as product swatches, as well as a null effect. The model is trained end-to-end in an unsupervised fashion. To condition a generator of the model, convolutional conditional batch normalization (CCBN) is used to apply the vector encoding the reference swatch images that represent the makeup properties.

System and method for augmented reality using conditional cycle-consistent generative image-to-image translation models

Presented is a convolutional neural network (CNN) model for fingernail tracking, and a method design for nail polish rendering. Using current software and hardware, the CNN model and method to render nail polish runs in real-time on both iOS and web platforms. A use of Loss Mean Pooling (LMP) coupled with a cascaded model architecture simultaneously enables pixel-accurate fingernail predictions at up to 640&#xd7;480 resolution. The proposed post-processing and rendering method takes advantage of the model's multiple output predictions to render gradients on individual fingernails, and to hide the light-colored distal edge when rendering on top of natural fingernails by stretching the nail mask in the direction of the fingernail tip. Teachings herein may be applied to track objects other than fingernails and to apply appearance effects other than color.

Growing community of inventors

Toronto, Canada

Edmund Phung