Amazon Technologies, Inc.

Han-Kai Hsu

Mohamed Kamal Omar

Xiaohang Sun

Ashutosh Sanan

Wentao Zhu

Yash Chaturvedi

Yujia Chen

Mauricio Alejandro Flores Rios

Linda Liu

Systems, methods, and computer-readable media are disclosed for systems and methods multimodal indexing of video using machine learning. An example method may include deceiving, by a video encoder of an audio-video transformer neural network comprising one or more computer processors coupled to memory, a first frame and a second frame associated with a first segment of a video. The example method may also include receiving, by an audio encoder of the audio-video transformer neural network, an audio spectrogram comprising first audio data associated with the first segment of the video. generating, by the video encoder, a first video embedding. The example method may also include generating, by the audio encoder, a first audio embedding. The example method may also include determining a fusion of the first video embedding and the first audio embedding using a multimodal bottleneck token. The example method may also include determining an output including the first video embedding and the first audio embedding. The example method may also include determining a classification of the first portion of the video based on the output.

Systems and methods for multimodal indexing of video using machine learning

Systems, methods, and computer-readable media are disclosed for determining virtual product placement opportunities in a media content and determining product candidates for virtual insertion into the media content. The product placement system may determine shot segments from the media content and for each shot segment may determine candidate product placement locations. The product placement system may determine contextual information from the shot segments and from the contextual information determine candidate products suitable for the product placement locations. The product placement system may determine total screen time for each product placement opportunity as well as quality of each opportunity. For each product and product placement opportunity, the product placement system may determine an expected revenue and a projected insertion cost.

Systems and methods for selecting product placement locations and products

Systems and techniques for retrieving video data associated with a selected attribute are described. The systems and techniques include receiving a multimodal input associated with the attribute for querying a catalog of video data to identify video data including the attribute. A first embedding is determined for the input using a first encoder to map the input to a representation space. A second embedding is determined for the video data to map the video data to the representation space. A similarity score is determined between the video data and the input based on a distance between the embeddings. The video data associated with the attribute may be selected based on the similarity score.

Automated content filtering using image retrieval models

Growing community of inventors

Seattle, WA, United States of America

Han-Kai Hsu