Classical and quantum algorithms for orthogonal neural networks

Abstract

Orthogonal neural networks impose orthogonality on the weight matrices. They may achieve higher accuracy and avoid evanescent or explosive gradients for deep architectures. Several classical gradient descent methods have been proposed to preserve orthogonality while updating the weight matrices, but these techniques suffer from long running times and provide only approximate orthogonality. In this disclosure, we introduce a new type of neural network layer. The layer allows for gradient descent with perfect orthogonality with the same asymptotic running time as a standard layer. The layer is inspired by quantum computing and can therefore be applied on a classical computing system as well as on a quantum computing system. It may be used as a building block for quantum neural networks and fast orthogonal neural networks.