Mixed-precision multiply-and-accumulation tree structure to maximize memory bandwidth usage for computational acceleration of generative large language model

Abstract

Provided is a mixed-precision multiply-and-accumulation (MAC) tree structure to maximize memory bandwidth usage for computational acceleration of a generative large language model. A MAC tree-based operator may include a plurality of floating-point (FP) multipliers connected in parallel and configured to process a multiplication operation on data delivered from an external memory; a plurality of first converters configured to convert output of each of the plurality of FP multipliers from floating point to fixed point; a fixed-point (FXP) adder tree connected to the plurality of first converters and configured to process summation of multiplication results of the plurality of FP multipliers; an FXP accumulator configured to accumulate output of the FXP adder tree; and a second converter configured to convert output of the FXP accumulator from the fixed point to the floating point.