Automatic pipeline for embedding vector generation

Abstract

A data processing service receives a request from a user. The request specifies a task for generating embedding vectors to represent a target dataset using a target embedding model. The service applies an automatic pipeline to the target dataset, which pre-processes the target dataset to generate batches of the target dataset. Each batch includes a plurality of data chunks in the target dataset. The automatic pipeline assigns computing resources to each batch of the target dataset; and applies, by the assigned computing resources, the target embedding model to each batch to generate the embedding vectors. Each embedding vector represents a data chunk of the target dataset in a latent space and is incrementally updated based on an update to a corresponding data chunk represented by the embedding vector. The configuration stores the generated one or more embedding vectors in a vector database.