Self-attention in homomorphic encryption deep learning architectures

Abstract

Mechanisms are provided for optimizing a deep learning (DL) computer model for homomorphic encryption (HE) workload processing. The mechanisms receive an original DL computer model architecture that is to be optimized for HE workload processing, and modifying the original DL computer model architecture by replacing a self-attention layer of the original DL computer model with an HE friendly self-attention layer that comprises a Power SoftMax function that does not have exponent terms, to thereby generate a modified DL computer model architecture. The mechanisms execute a machine learning training of the modified DL computer model architecture, approximate one or more elements of the Power SoftMax function with polynomials to generate a trained HE optimized DL computer model, and output the trained HE optimized DL computer model for execution on HE workloads.