Adobe, Inc.

Microsoft Technology Licensing, LLC

Joon-Young Lee

Kalyan Krishna Sunkavalli

Zhe Lin

Xiaohui Shen

Byungmoon Kim

Seoungwug Oh

Jinwoong Jung

Hailin Jin

Jianming Zhang

Sunil Sharadchandra Hadap

Zhuo Hui

Gavin Miller

Seoung Wug Oh

Brian Lynn Price

Zhaowen Wang

Ning Xu

Bryan Russell

Markus Woodson

Sanghyun Woo

Ruppesh Nalwaya

Seonguk Seo

Jingwan Lu

Xin Lu

Li-Yi Wei

Jonathan Eisenmann

Yasuyuki Matsushita

Haoxiang Li

Qi Sun

John Nelson

Fabian David Caba Heilbron

Hao Hu

Dahun Kim

Jaedong Hwang

Xuaner Zhang

Xin Ye

Kwanggyoon Seo

Kwanyong Park

Hongje Seong

Wujun Wang

Adam Pikielny

Rameswar Panda

Wei-Chih Hung

Embodiments are disclosed for performing per-clip object segmentation of objects in a video sequence using machine learning. In particular, in one or more embodiments, the disclosed systems and methods comprise receiving a query video sequence and memory data, the memory data including a memory video frame from the query video sequence and an annotated memory video frame including an object mask for an object in the memory video frame, segmenting the query video sequence into a plurality of query video clips and passing a first set of query video frames of a first query video clip and the memory data through a trained encoder-decoder network, predicting a modified set of query video frames, including predictions of object masks for the object, and updating the memory data to include one or more frames of the first set of query video frames and the modified set of query video frames.

Per-clip video object segmentation using machine learning

In implementations of systems for joint trimap estimation and alpha matte prediction, a computing device implements a matting system to estimate a trimap for a frame of a digital video using a first stage of a machine learning model. An alpha matte is predicted for the frame based on the trimap and the frame using a second stage of the machine learning model. The matting system generates a refined trimap and a refined alpha matte for the frame based on the alpha matte, the trimap, and the frame using a third stage of the machine learning model. An additional trimap is estimated for an additional frame of the digital video based on the refined trimap and the refined alpha matte using the first stage of the machine learning model.

Joint trimap estimation and alpha matte prediction for video matting

Embodiments are disclosed for a machine learning-based chroma keying process. The method may include receiving an input including an image depicting a chroma key scene and a color value corresponding to a background color of the image. The method may further include generating a preprocessed image by concatenating the image and the color value. The method may further include providing the preprocessed image to a trained neural network. The method may further include generating, using the trained neural network, an alpha matte representation of the image based on the preprocessed image.

Machine learning-based chroma keying process

Systems and methods for video processing are described. Embodiments of the present disclosure identify an image that depicts an expression of a face; encode the image to obtain a latent code representing the image; edit the latent code to obtain an edited latent code that represents the face with a target attribute that is different from an original attribute of the face and with an edited expression that is different from the expression of the face; modify the edited latent code to obtain a modified latent code that represents the face with the target attribute and a modified expression, wherein a difference between the expression and the modified expression is less than a difference between the expression and the edited expression; and generate a modified image based on the modified latent code, wherein the modified image depicts the face with the target attribute and the modified expression.

Editing portrait videos

Systems, methods, and non-transitory computer-readable media are disclosed for automatic tagging of videos. In particular, in one or more embodiments, the disclosed systems generate a set of tagged feature vectors (e.g., tagged feature vectors based on action-rich digital videos) to utilize to generate tags for an input digital video. For instance, the disclosed systems can extract a set of frames for the input digital video and generate feature vectors from the set of frames. In some embodiments, the disclosed systems generate aggregated feature vectors from the feature vectors. Furthermore, the disclosed systems can utilize the feature vectors (or aggregated feature vectors) to identify similar tagged feature vectors from the set of tagged feature vectors. Additionally, the disclosed systems can generate a set of tags for the input digital videos by aggregating one or more tags corresponding to identified similar tagged feature vectors.

Generating action tags for digital videos

This disclosure describes one or more implementations of a panoptic segmentation system that generates panoptic segmented digital images that classify both known and unknown instances of digital images. For example, the panoptic segmentation system builds and utilizes a panoptic segmentation neural network to discover, cluster, and segment new unknown object subclasses for previously unknown object instances. In addition, the panoptic segmentation system can determine additional unknown object instances from additional digital images. Moreover, in some implementations, the panoptic segmentation system utilizes the newly generated unknown object subclasses to refine and tune the panoptic segmentation neural network to improve the detection of unknown object instances in input digital images.

Generating improved panoptic segmented digital images based on panoptic segmentation neural networks that utilize exemplar unknown object classes

Methods and systems are provided for performing material capture to determine properties of an imaged surface. A plurality of images can be received depicting a material surface. The plurality of images can be calibrated to align corresponding pixels of the images and determine reflectance information for at least a portion of the aligned pixels. After calibration, a set of reference materials from a material library can be selected using the calibrated images. The set of reference materials can be used to determine a material model that accurately represents properties of the material surface.

Material capture using imaging

A method, apparatus, and non-transitory computer readable medium for referring image segmentation are described. Embodiments of the method, apparatus, and non-transitory computer readable medium may extract an image feature vector from an input image, extract a plurality of language feature vectors for a referral expression, wherein each of the plurality of language feature vectors comprises a different number of dimensions, combine each of the language feature vectors with the image feature vector using a fusion module to produce a plurality of self-attention vectors, combine the plurality of self-attention vectors to produce a multi-modal feature vector, and decode the multi-modal feature vector to produce an image mask indicating a portion of the input image corresponding to the referral expression.

Referring image segmentation

Systems and methods for panoptic video segmentation are described. A method may include identifying a target frame and a reference frame from a video, generating target features for the target frame and reference features for the reference frame, combining the target features and the reference features to produce fused features for the target frame, generating a feature matrix comprising a correspondence between objects from the reference features and objects from the fused features; and generating panoptic segmentation information for the target frame based on the feature matrix.

Video panoptic segmentation

Systems and methods for video object segmentation are described. Embodiments of systems and methods may receive a referral expression and a video comprising a plurality of image frames, generate a first image mask based on the referral expression and a first image frame of the plurality of image frames, generate a second image mask based on the referral expression, the first image frame, the first image mask, and a second image frame of the plurality of image frames, and generate annotation information for the video including the first image mask overlaid on the first image frame and the second image mask overlaid on the second image frame.

Unified referring video object segmentation network

Systems and techniques are described that provide for generalizable approach policy learning and implementation for robotic object approaching. Described techniques provide fast and accurate approaching of a specified object, or type of object, in many different environments. The described techniques enable a robot to receive an identification of an object or type of object from a user, and then navigate to the desired object, without further control from the user. Moreover, the approach of the robot to the desired object is performed efficiently, e.g., with a minimum number of movements. Further, the approach techniques may be used even when the robot is placed in a new environment, such as when the same type of object must be approached in multiple settings.

Generalizable robot approach control techniques

Certain aspects involve using a space-time memory network to locate one or more target objects in video content for segmentation or other object classification. In one example, a video editor generates a query key map and a query value map by applying a space-time memory network to features of a query frame from video content. The video editor retrieves a memory key map and a memory value map that are computed, with the space-time memory network, from a set of memory frames from the video content. The video editor computes memory weights by applying a similarity function to the memory key map and the query key map. The video editor classifies content in the query frame as depicting the target feature using a weighted summation that includes the memory weights applied to memory locations in the memory value map.

Space-time memory network for locating target object in video content

Various embodiments describe video object segmentation using a neural network and the training of the neural network. The neural network both detects a target object in the current frame based on a reference frame and a reference mask that define the target object and propagates the segmentation mask of the target object for a previous frame to the current frame to generate a segmentation mask for the current frame. In some embodiments, the neural network is pre-trained using synthetically generated static training images and is then fine-tuned using training videos.

Video object segmentation by reference-guided mask propagation

Generating tags for a digital video

Embodiments herein describe a framework for classifying images. In some embodiments, it is determined whether an image includes synthetic image content. If it does, characteristics of the image are analyzed to determine if the image includes characteristics particular to panoramic images (e.g., possess a threshold equivalency of pixel values among the top and/or bottom boundaries of the image, or a difference between summed pixel values of the pixels comprising the right vertical boundary of the image and summed pixel values of the pixels comprising the left vertical boundary of the image being less than or equal to a threshold value). If the image includes characteristics particular to panoramic images, the image is classified as a synthetic panoramic image. If the image is determined to not include synthetic image content, a neural network is applied to the image and the image is classified as one of non-synthetic panoramic or non-synthetic non-panoramic.

Classifying panoramic images

Growing community of inventors

Milpitas, CA, United States of America

Joon-Young Lee