Multimedia data search using multi-modal feature embeddings

Abstract

Aspects of the disclosed technology provide solutions for searching objects within multimedia content based on multi-modal embeddings. An example method can include receiving media content including a plurality of video frames. The method can include steps for generating, using a pre-output layer of a machine learning algorithm, one or more multimodal feature embeddings describing at least one object for the plurality of video frames, receiving a query including a request to search the media content for a matching object, determining whether the media content includes the matching object based on the one or more multimodal feature embeddings describing the at least one object, and returning one or more results in response to determining that the media content includes the matching object. Systems and machine-readable media are also provided.