Using fixed-weight language models to create and interact with a retrieval index

Abstract

A technique uses an encoder system to produce an index of target item embeddings. Each target item embedding is input-agnostic and universal in the sense that different expressions of a target concept, produced using different combinations of input modes, map to the same target item embedding in the index. The encoder system throttles the amount of computations it performs based on the assessed capabilities of an execution platform. A retrieval system processes a multimodal input query by first generating a candidate set of target item embeddings in the index that match the input query, and then using a filtering operation to identify those target item embeddings that are most likely to match the input query. The encoder system and the retrieval system rely on language-based components having weights that are held constant during a training operation. Other weights of these systems are updated during the training operation.