Techniques for selecting between scalar quantization and trellis coded quantization when encoding video data

Abstract

In various embodiments, an encoder receives transform coefficients of prediction residues associated with video data. The encoder sets a flag value based on at least one of metadata or transform statistics associated with a vector of the transform coefficients. The encoder performs trellis coded quantization operations on the vector of the transform coefficients to generate a vector of quantization indices or one or more scalar quantization operations on the transform coefficients to generate a set of quantization indices based on the flag value. The encoder performs entropy coding operations on either the vector of quantization indices or a vector corresponding to the set of quantization indices to generate encoded video data.