Multi-scale transformer for image analysis

Abstract

The technology employs a patch-based multi-scale Transformer () that is usable with various imaging applications. This avoids constraints on image fixed input size and predicts the quality effectively on a native resolution image. A native resolution image () is transformed into a multi-scale representation (), enabling the Transformer's self-attention mechanism to capture information on both fine-grained detailed patches and coarse-grained global patches. Spatial embedding () is employed to map patch positions to a fixed grid, in which patch locations at each scale are hashed to the same grid. A separate scale embedding () is employed to distinguish patches coming from different scales in the multiscale representation. Self-attention () is performed to create a final image representation. In some instances, prior to performing self-attention, the system may prepend a learnable classification token () to the set of input tokens.