Beyond shared hierarchies: deep multitask learning through soft layer ordering

Abstract

The technology disclosed identifies parallel ordering of shared layers as a common assumption underlying existing deep multitask learning (MTL) approaches. This assumption restricts the kinds of shared structure that can be learned between tasks. The technology disclosed demonstrates how direct approaches to removing this assumption can ease the integration of information across plentiful and diverse tasks. The technology disclosed introduces soft ordering as a method for learning how to apply layers in different ways at different depths for different tasks, while simultaneously learning the layers themselves. Soft ordering outperforms parallel ordering methods as well as single-task learning across a suite of domains. Results show that deep MTL can be improved while generating a compact set of multipurpose functional primitives, thus aligning more closely with our understanding of complex real-world processes.