Snap Inc.

Ju Hu

Yanyu Li

Sergey Tulyakov

Dhritiman Sagar

Jian Ren

Aleksei Stoliar

Georgios Evangelidis

Yang Wen

Vladislav Shakhrai

Qing Jin

Huan Wang

Geng Yuan

Pavlo Chemerys

Junli Cao

Ergeta Muca

Colin Eles

A neural light field (NeLF) that runs real-time on mobile devices for neural rendering of three dimensional (3D) scenes, referred to as MobileR2L. The MobileR2L architecture runs efficiently on mobile devices with low latency and small size, and it achieves high-resolution generation while maintaining real-time inference for both synthetic and real-world 3D scenes on mobile devices. The MobileR2L has a network backbone including a convolutional layer embedding an input image at a resolution, residual blocks uploading the embedded image, and super-resolution modules receiving the uploaded embedded image and rendering an output image having a higher resolution than the embedded image. The convolution layer generates a number of rays equal to a number of pixels in the input image, where a partial number of the rays is uploaded to the super-resolution modules.

Real-time neural light field on mobile devices

Described is a system for improving machine learning models. In some cases, the system improves such models by identifying a performance characteristic for machine learning model blocks in an iterative denoising process of a machine learning model, connecting a prior machine learning model block with a subsequent machine learning model block of the machine learning model blocks within the machine learning model based on the identified performance characteristic, identifying a prompt of a user, the prompt indicative of an intent of the user for generative images, and analyzing data corresponding to the prompt using the machine learning model to generate one or more images, the machine learning model trained to generate images based on data corresponding to prompts.

Text-to-image diffusion model rearchitecture

A vision transformer network having extremely low latency and usable on mobile devices, such as smart eyewear devices and other augmented reality (AR) and virtual reality (VR) devices. The transformer network processes an input image, and the network includes a convolution stem configured to patch embed the image. A first stack of stages including at least two stages of 4-Dimension (4D) metablocks (MBs) (MB) follow the convolution stem. A second stack of stages including at least two stages of 3-Dimension MBs (MB) follow the MBstages. Each of the MBstages and each of the MBstages include different layer configurations, and each of the MBstages and each of the MBstages include a token mixer. The MBstages each additionally include a multi-head self attention (MHSA) processing block.

Efficientformer vision transformer

Growing community of inventors

Los Angeles, CA, United States of America

Ju Hu