Amazon Technologies, Inc.

Ashutosh Sanan

Mohamed Kamal Omar

Xiaohang Sun

Han-Kai Hsu

Wentao Zhu

Systems, methods, and computer-readable media are disclosed for systems and methods multimodal indexing of video using machine learning. An example method may include deceiving, by a video encoder of an audio-video transformer neural network comprising one or more computer processors coupled to memory, a first frame and a second frame associated with a first segment of a video. The example method may also include receiving, by an audio encoder of the audio-video transformer neural network, an audio spectrogram comprising first audio data associated with the first segment of the video. generating, by the video encoder, a first video embedding. The example method may also include generating, by the audio encoder, a first audio embedding. The example method may also include determining a fusion of the first video embedding and the first audio embedding using a multimodal bottleneck token. The example method may also include determining an output including the first video embedding and the first audio embedding. The example method may also include determining a classification of the first portion of the video based on the output.

Systems and methods for multimodal indexing of video using machine learning

Systems and techniques for retrieving video data associated with a selected attribute are described. The systems and techniques include receiving a multimodal input associated with the attribute for querying a catalog of video data to identify video data including the attribute. A first embedding is determined for the input using a first encoder to map the input to a representation space. A second embedding is determined for the video data to map the video data to the representation space. A similarity score is determined between the video data and the input based on a distance between the embeddings. The video data associated with the attribute may be selected based on the similarity score.

Growing community of inventors

Seattle, WA, United States of America

Ashutosh Sanan