Reducing power consumption by hardware accelerator during generation and transmission of machine learning inferences

Abstract

A hardware accelerator can receive, from a host processor, a slice of input data at a time-step. The hardware accelerator can process the input data using a machine learning model deployed on the hardware accelerator to compute a respective probability among multiple probabilities for each of multiple classes. The respective probability for each class being a likelihood that content in the slice belongs to the class. The hardware accelerator can determine, from the multiple probabilities, a preset number of highest probabilities for the slice of input data. The hardware accelerator can transmit the preset number of highest probabilities for the slice to the host processor. Related apparatus, systems, techniques and articles are also described.