L'oreal

Elmoznino, Eric

Flament, Frederic Antoinin Raymond Serge

Jiang, Ruowei

Kezele, Irina

Ma, Junwei

Levinshtein, Alex

Despois, Julien

Aarabi, Parham

Ma, He

Perrot, Matthieu

Ruowei Jiang

Parham Aarabi

Irina Kezele

Alex Levinshtein

Eric Elmoznino

Frederic Antoinin Raymond Serge Flament

Julien Despois

Edmund Phung

Matthieu Perrot

Cheng Chang

Zhi Yu

Junwei Ma

Yuze Zhang

He Ma

Zeqi Li

Wenzhangzhi Guo

Brendan Duke

Jeffrey Houghton

Sophie Seite

Mathieu Perrot

Cong Wei

Vision Transformers (ViT) have shown their competitive advantages performance-wise compared to convolutional neural networks (CNNs) though they often come with high computational costs. Methods, systems and techniques herein learn instance-dependent attention patterns, utilizing a lightweight connectivity predictor module to estimate a connectivity score of each pair of tokens. Intuitively, two tokens have high connectivity scores if the features are considered relevant either spatially or semantically. As each token only attends to a small number of other tokens, the binarized connectivity masks are often very sparse by nature providing an opportunity to accelerate the network via sparse computations. Equipped with the learned unstructured attention pattern, sparse attention ViT produces a superior Pareto-optimal trade-off between FLOPs and top-1 accuracy on ImageNet compared to token sparsity (48%˜69% FLOPs reduction of MHSA; accuracy drop within 0.4%). Combining attention and token sparsity reduces VIT FLOPs by over 60%.

Systems, methods and techniques for learning and using sparse instance-dependent attention for efficient vision transformers

Systems, methods and techniques provide for acne localization, counting and visualization. An image is processed using a trained model to identify objects. The model may be a deep learning (e.g. convolutional neural) network configured for object classification with a detection focus on small objects. The image may be a frontal or profile facial image, processed end to end. The model identifies and localizes different types of acne. Instances are counted and visualized such as by annotating the source image. An example annotation is an overlay identifying a type and location of each instance. Counts by acne type assist with scoring. A product and/or service may be recommended in response to the identification of the acne (e.g. the type, localization, counting and/or a score).

Systems and methods for acne counting, localization and visualization

There is provided methods, devices and techniques to process an image using a deep learning model to achieve continuous effect simulation by a unified network where a simple (effect class) estimator is embedded into a regular encoder-decoder architecture. The estimator allows learning of model-estimated class embeddings of all effect classes (e.g. progressive degrees of the effect), thus representing the continuous effect information without manual efforts in selecting proper anchor effect groups. In an embodiment, given a target age class, there is derived a personalized age embedding which considers two aspects of face aging: 1) a personalized residual age embedding at a model-estimated age of the subject, preserving the subject's aging information; and 2) exemplar-face aging basis at the target age, encoding the shared aging patterns among the entire population. Training and runtime (inference time) embodiments are described including an AR application that generates recommendations and provides ecommerce services.

Applying a continuous effect via model-estimated class embeddings

GANs based generators are useful to perform image to image translations. GANs models have large storage sizes and resource use requirements such that they are too large to be deployed directly on mobile devices. Systems and methods define through conditioning a student GANs model having a student generator that is scaled downwardly from a teacher GANs model (and generator) using knowledge distillation. A semantic relation knowledge distillation loss is used to transfer semantic knowledge from an intermediate layer of the teacher to an intermediate layer of the student. Student generators thus defined are stored and executed by mobile devices such as smartphones and laptops to provide augmented reality experiences. Effects are simulated on images, including makeup, hair, nail and age simulation effects.

Semantic relation preserving knowledge distillation for image-to-image translation

There is described a deep learning supervised regression based model including methods and systems for facial attribute prediction and use thereof. An example of use is an augmented and/or virtual reality interface to provide a modified image responsive to facial attribute predictions determined from the image. Facial effects matching facial attributes are selected to be applied in the interface.

Systems and methods for improved facial attribute classification and use thereof

Systems and methods process images to determine a skin condition severity analysis and to visualize a skin analysis such as using a deep neural network (e.g. a convolutional neural network) where a problem was formulated as a regression task with integer-only labels. Auxiliary classification tasks (for example, comprising gender and ethnicity predictions) are introduced to improve performance. Scoring and other image processing techniques may be used (e.g. in assoc. with the model) to visualize results such as highlighting the analyzed image. It is demonstrated that the visualization of results, which highlight skin condition affected areas, can also provide perspicuous explanations for the model. A plurality (k) of data augmentations may be made to a source image to yield k augmented images for processing. Activation masks (e.g. heatmaps) produced from processing the k augmented images are used to define a final map to visualize the skin analysis.

Systems and methods to process images for skin analysis and to visualize skin analysis

A system and method implement deep learning on a mobile device to provide a convolutional neural network (CNN) for real time processing of video, for example, to color hair. Images are processed using the CNN to define a respective hair matte of hair pixels. The respective object mattes may be used to determine which pixels to adjust when adjusting pixel values such as to change color, lighting, texture, etc. The CNN may comprise a (pre-trained) network for image classification adapted to produce the segmentation mask. The CNN may be trained for image segmentation (e.g. using coarse segmentation data) to minimize a mask-image gradient consistency loss. The CNN may further use skip connections between corresponding layers of an encoder stage and a decoder stage where shallower layers in the encoder, which contain high-res but weak features are combined with low resolution but powerful features from deeper decoder layers.

System and method for image processing using deep neural networks

There is shown and described a deep learning based system and method for skin diagnostics as well as testing metrics that show that such a deep learning based system outperforms human experts on the task of apparent skin diagnostics. Also shown and described is a system and method of monitoring a skin treatment regime using a deep learning based system and method for skin diagnostics.

Automatic image-based skin diagnostics using deep learning

Growing community of inventors

Toronto, Canada

Ruowei Jiang