Computer-implemented methods for machine learning model based spatial-temporal adaptive shift for end-to-end text-video retrieval

Abstract

Techniques for performing a machine learning model based spatial-temporal adaptive shift for end-to-end text-video retrieval are described. According to some examples, a computer-implemented method includes receiving a video comprising a plurality of frames at a content delivery service; generating, by the content delivery service, a set of embeddings for each of a plurality of sections of each frame of the plurality of frames; determining, by a candidate selector machine learning model of the content delivery service, a proper subset of the plurality of sections of each frame of the plurality of frames for a time shift based on the set of embeddings; time shifting, by the content delivery service, the proper subset of the plurality of sections of each frame of the plurality of frames to generate time shifted frames; generating, by the content delivery service, an updated set of embeddings based on the time shifted frames; receiving a search request comprising input text from a user device; determining the video is a match for the search request based on the input text and the updated set of embeddings for the time shifted frames; and sending the video to the user device based on the match.