Transformer-based text encoder for passage retrieval

Abstract

A computing system includes a logic subsystem and a storage subsystem holding instructions executable by the logic subsystem to implement a transformer-based text encoder. The transformer-based text encoder includes a plurality of transformer blocks previously-trained to apply encoding operations to computer-readable text representations of input text strings, the computer-readable text representations including computer-readable question representations of input text questions, and computer-readable passage representations of input text passages. The plurality of transformer blocks include a shared transformer block trained for both the computer-readable question representations and the computer-readable passage representations and a specialized transformer block including two or more input-specific subnetworks, and a routing function to select an input-specific subnetwork of the two or more input-specific subnetworks for each of the computer-readable text representations.