Microsoft Technology Licensing, LLC

Dongdong Chen

Lu Yuan

Yinpeng Chen

Jing Liao

Xiyang Dai

Mengchen Liu

Gang Hua

Lei Zhang

Jianmin Bao

Zicheng Liu

Fang Wen

Jingdong Wang

Mei Chen

Ye Yu

Qiankun Liu

Bin Xiao

Mingming He

Yunsheng Li

Xiaoyi Dong

Dong Chen

Ting Zhang

Dengpan Fu

The disclosure herein describes training an encoder network to inpaint images with masked portions. A primary encoding process is used to encode a visible portion of a masked input image into encoded token data. The encoded token data is then decoded into both pixel regression output and feature prediction output, wherein both outputs include inpainted image data associated with the masked portion of the masked input image. A pixel regression loss is determined using the pixel regression output and pixel data of an unmasked version of the masked input image. A feature prediction loss is determined using the feature prediction output and ground truth encoding output of the unmasked version of the masked input image. The primary encoding process is then trained using the pixel regression loss and the feature prediction loss, whereby the primary encoding process is trained to encode structural features of input images into encoded token data.

Training masked autoencoders for image inpainting

A computer device for automatic feature detection comprises a processor, a communication device, and a memory configured to hold instructions executable by the processor to instantiate a dynamic convolution neural network, receive input data via the communication network, and execute the dynamic convolution neural network to automatically detect features in the input data. The dynamic convolution neural network compresses the input data from an input space having a dimensionality equal to a predetermined number of channels into an intermediate space having a dimensionality less than the number of channels. The dynamic convolution neural network dynamically fuses the channels into an intermediate representation within the intermediate space and expands the intermediate representation from the intermediate space to an expanded representation in an output space having a higher dimensionality than the dimensionality of the intermediate space. The features in the input data are automatically detected based on the expanded representation.

Dynamic matrix convolution with channel fusion

A system for tracking a target object across a plurality of image frames. The system comprises a logic machine and a storage machine. The storage machine holds instructions executable by the logic machine to calculate a trajectory for the target object over one or more previous frames occurring before a target frame. Responsive to assessing no detection of the target object in the target frame, the instructions are executable to predict an estimated region for the target object based on the trajectory, predict an occlusion center based on a set of candidate occluding locations for a set of other objects within a threshold distance of the estimated region, each location of the set of candidate occluding locations overlapping with the estimated region, and automatically estimate a bounding box for the target object in the target frame based on the occlusion center.

Occlusion-aware multi-object tracking

The disclosure herein describes generating an inpainted image from a masked image using a patch-based encoder and an unquantized transformer. An image including a masked region and an unmasked region is received, and the received image is divided into a plurality of patches including masked patches. The plurality of patches is encoded into a plurality of feature vectors, wherein each patch is encoded to a feature vector. Using a transformer, a predicted token is generated for each masked patch using a feature vector encoded from the masked patch, and a quantized vector of the masked patch is determined using generated predicted token and a masked patch-specific codebook. The determined quantized vector of the masked patch is included into a set of quantized vectors associated with the plurality of patches, and an output image is generated from the set of quantized vectors using a decoder.

Generating an inpainted image from a masked image using a patch-based encoder

Systems and methods for object detection generate a feature pyramid corresponding to image data, and rescaling the feature pyramid to a scale corresponding to a median level of the feature pyramid, wherein the rescaled feature pyramid is a four-dimensional (4D) tensor. The 4D tensor is reshaped into a three-dimensional (3D) tensor having individual perspectives including scale features, spatial features, and task features corresponding to different dimensions of the 3D tensor. The 3D tensor is used with a plurality of attention layers to update a plurality of feature maps associated with the image data. Object detection is performed on the image data using the updated plurality of feature maps.

Dynamic head for object detection

A computer-vision method includes recognizing a feature representation of a query image depicting an unknown subject. A similarity score is computed between the representation of the query image and feature representations of a plurality of gallery images collectively depicting two or more different subjects with at least two or more gallery images for each subject, and each gallery image having a label identifying which of the subjects is depicted. One or more updated feature representations of the query image are sequentially iterated based on one or more of the computed similarity scores. For each of the one or more updated feature representations, an updated similarity score is computed between the updated feature representation and the feature representations of each of the gallery images. The unknown subject is identified based on a gallery image having a highest updated similarity score.

Subject identification based on iterated feature representation

Image stylization is based on a learning network. A learning network is trained with a plurality of images and a reference image with a particular texture style. A plurality of different sub-networks of the learning network is trained, respectively. Specifically, one of the sub-networks is trained to extract one or more feature maps from the source image and transform the feature maps with the texture style applied thereon to a target image. Each of the feature maps indicates part of feature information of the source image. Another sub-network is trained to apply a specified texture style to the extracted feature maps, such that the target image generated based on the processed feature maps can embody the specified texture style.

Image stylization based on learning network

According to implementations of the subject matter described herein, there is provided an image colorization solution. The solution includes determining a similarity between contents presented in a grayscale source image and a color reference image and determining a col or target image corresponding to the source image based on the similarity. Specifically, a first and a second sets of blocks similar and dissimilar to the reference image are determined based on the similarity; a first color for the first set of blocks is determined based on a color of corresponding blocks in the reference image; a second color for the second set of blocks is determined independently of the reference image. Through this solution, it is possible to provide user controllability and customized effects in colorization, and there is no strict requirement on correspondence between the color image and grayscale image, achieving more robustness to selection of color reference images.

Image colorization based on reference information

In accordance with implementations of the subject matter described herein, there is proposed a solution of visual stylization of stereoscopic images. In the solution, a first feature map for a first source image and a second feature map for a second source image are extracted. The first and second source images correspond to first and second views of a stereoscopic image, respectively. A first unidirectional disparity from the first source image to the second source image is determined based on the first and second source images. First and second target images having a specified visual style are generated by processing the first and second feature maps based on the first unidirectional disparity. Through the solution, a disparity between two source images of a stereoscopic image are taken into account when performing the visual style transfer, thereby maintaining a stereoscopic effect in the stereoscopic image consisting of the target images.

Growing community of inventors

Redmond, WA, United States of America

Dongdong Chen