Method and apparatus for static weight permutation in transformer workloads

Abstract

A method of operating a transformer compute apparatus using weight permutations. The method includes performing a permutation process on both a target weight matrix known to have one or more outlier values and a matching permutation process on an associated weight matrix of a target workload operation. This target weight matrix can be permuted across a quantization dimension according to its vector norms in ascending or descending order. To preserve the output of the operation, the other weight matrix is permuted to match the permutation of the target weight matrix, as the synchronized permutation of both weight matrices will result in the same product of those matrices. The synchronized permutation of both matrices reduces the variance of the resulting output, which also reduces variance of the elements within each quantized unit of the resulting output in the quantization format and thus mitigates the impact of the outlier values.