Google Inc.

Qifei Wang

Yilin Wang

Peyman Milanfar

Feng Yang

Junjie Ke

Ming-Hsuan Yang

Balineedu Adsumilli

Neil Aylon Charles Birkbeck

Boqing Gong

Deqing Sun

Taihong Xiao

Jinwei Yuan

Miao Yin

A method for generating, for a video stream of a first spatial resolution and a first temporal resolution, a first reduced quality steam of a second spatial resolution and a second reduced-quality stream of a second temporal resolution. A first subset of STPs is sampled from the first reduced-quality stream and a second subset of STPs is sampled from the second reduced-quality stream. Using a machine learning model (MLM) the STPs are processed to identify a quality score for each quality-representative STPs that are representative of a quality of the video stream. One or more quality-improving actions for the video stream are identified using the quality scores of the quality-representative STPs.

Highly efficient model for video quality assessment

The technology employs a patch-based multi-scale Transformer () that is usable with various imaging applications. This avoids constraints on image fixed input size and predicts the quality effectively on a native resolution image. A native resolution image () is transformed into a multi-scale representation (), enabling the Transformer's self-attention mechanism to capture information on both fine-grained detailed patches and coarse-grained global patches. Spatial embedding () is employed to map patch positions to a fixed grid, in which patch locations at each scale are hashed to the same grid. A separate scale embedding () is employed to distinguish patches coming from different scales in the multiscale representation. Self-attention () is performed to create a final image representation. In some instances, prior to performing self-attention, the system may prepend a learnable classification token () to the set of input tokens.

Multi-scale transformer for image analysis

A method includes obtaining a first plurality of feature vectors associated with a first image and a second plurality of feature vectors associated with a second image. The method also includes generating a plurality of transformed feature vectors by transforming each respective feature vector of the first plurality of feature vectors by a kernel matrix trained to define an elliptical inner product space. The method additionally includes generating a cost volume by determining, for each respective transformed feature vector of the plurality of transformed feature vectors, a plurality of inner products, wherein each respective inner product of the plurality of inner products is between the respective transformed feature vector and a corresponding candidate feature vector of a corresponding subset of the second plurality of feature vectors. The method further includes determining, based on the cost volume, a pixel correspondence between the first image and the second image.

Learnable cost volume for determining pixel correspondence

Growing community of inventors

Sunnyvale, CA, United States of America

Qifei Wang