Amazon Technologies, Inc.

Xiaohang Sun

Mohamed Kamal Omar

Muhammad Raffay Hamid

Ahmed Aly Saad Ahmed

Han-Kai Hsu

Alexander Ratnikov

Travis Silvers

Hanxiao Deng

Tai-Ching Li

Ashutosh Sanan

Wentao Zhu

Ivan Ryndin

Systems, methods, and computer-readable media are disclosed for systems and methods multimodal indexing of video using machine learning. An example method may include deceiving, by a video encoder of an audio-video transformer neural network comprising one or more computer processors coupled to memory, a first frame and a second frame associated with a first segment of a video. The example method may also include receiving, by an audio encoder of the audio-video transformer neural network, an audio spectrogram comprising first audio data associated with the first segment of the video. generating, by the video encoder, a first video embedding. The example method may also include generating, by the audio encoder, a first audio embedding. The example method may also include determining a fusion of the first video embedding and the first audio embedding using a multimodal bottleneck token. The example method may also include determining an output including the first video embedding and the first audio embedding. The example method may also include determining a classification of the first portion of the video based on the output.

Systems and methods for multimodal indexing of video using machine learning

Systems and techniques for retrieving video data associated with a selected attribute are described. The systems and techniques include receiving a multimodal input associated with the attribute for querying a catalog of video data to identify video data including the attribute. A first embedding is determined for the input using a first encoder to map the input to a representation space. A second embedding is determined for the video data to map the video data to the representation space. A similarity score is determined between the video data and the input based on a distance between the embeddings. The video data associated with the attribute may be selected based on the similarity score.

Automated content filtering using image retrieval models

Techniques are disclosed for detecting an uncovered portion of a body of a person in a frame of video content. In an example, a first machine learning model of a computing system may output a first score for the frame based on a map that identifies a region of the frame associated with an uncovered body part type. Depending on a value of the first score, a second machine learning model that includes a neural network architecture may further analyze the frame to output a second score. The first score and second score may be merged to produce a third score for the frame. A plurality of scores may be determined, respectively, for frames of the video content, and a maximum score may be selected. The video content may be selected for presentation on a display for further evaluation based on the maximum score.

Frame-based body part detection in video clips

Techniques are disclosed for detecting a type of prurient activity shown by a portion of video content. In an example, a machine learning model of a computer system may receive a second portion of video content, the machine learning model including a neural network that is trained to analyze a temporal dimension of the second portion. The machine learning model determines a score indicating a likelihood that the video content shows the type of prurient activity based in part on applying a three-dimensional filter to the second portion of the video content. The computer system then generates a video clip that includes at least the portion of the video content showing the type of prurient activity based on the score, and provides the video clip for display.

Detecting prurient activity in video content

Growing community of inventors

Seattle, WA, United States of America

Xiaohang Sun