Training models under resource constraints for cross-device federated learning

Abstract

A system and a computer-implemented method of training a global student model is disclosed. The global student model and a teacher model are stored on a server and each include a first layer. The method includes transmitting local student models based on the global student model, the local student models each including an embedding layer and a first layer. The method includes receiving an embedding layer output of one of the local student models. The method includes performing a forward pass on the first layer of the teacher model, with the embedding layer output as an input, to generate a teacher model first layer output. The method includes transmitting the teacher model first layer output. The method includes receiving first layer weights of the local student models. The method includes calculating first layer weights of the global student model using the received first layer weights of the local student models.