Amazon Technologies, Inc.

Shixing Chen

Muhammad Raffay Hamid

Xiaohan Nie

Dongqing Zhang

Vimal Bhat

Kewen Chen

Tu Anh Ho

Xiang Hao

Prabhakar Gupta

Honey Gupta

Shiva Krishnamurthy

David Jiatian Fan

Alexander Andonian

Systems, devices, and methods are provided for searchability and discoverability of contextually relevant frames within digital content. Digital content, such as videos, may be segmented to identify a plurality of shots. Discoverability may be performed by identifying key frames of the digital content and using a contrastive language-image pre-training (CLIP) model to determine contextual relevance of a frame or shot to textual information associated with the digital content. Searchability may be performed by receiving search parameters and applying various filters to digital content to identify frames or shots that satisfy a user's search query.

Searchability and discoverability of contextually relevant frames within digital content

Soft information can be used to train an artificial intelligence (AI) model. For example, a computer system determines first training data that includes a first image, first text, a second image, second text, a first indication of a first correspondence between the first image and the first text, and a second indication of a second correspondence between the second image and the second text. The computer system determines a likelihood of the first image and the second text corresponding to each other. The computer system generates second training data that includes the first image, the first text, the second image, the second text, the likelihood instead of the first indication, and the second indication. The computer system trains an AI model based at least in part on the second training data.

Training an artificial intelligence model based on soft information

A plurality of similar video pairs may be determined based on one or more similarity information types. Each video pair of the plurality of similar video pairs may include a first respective video and a second respective video. For each video pair, one or more similar scene pairs may be determined. Each of the one or more similar scene pairs may include a respective first scene from the first respective video and a second respective scene from the second respective video. An encoder may be trained using a contrastive learning model that contrasts a plurality of similar scene pairs with a plurality of random scenes. The plurality of similar scene pairs may include the one or more scene pairs for each video pair. One or more scene features of one or more other scenes of one or more other videos may be determined using the encoder.

Contrastive learning of scene representation guided by video similarities

Techniques for automatic scene change detection are described. As one example, a computer-implemented method includes receiving a request to train an ensemble of machine learning models on a training dataset of videos having labels that indicate scene changes to detect a scene change in a video, partitioning each video file of the training dataset of videos into a plurality of shots, training the ensemble of machine learning models into a trained ensemble of machine learning models based at least in part on the plurality of shots of the training dataset of videos and the labels that indicate scene changes, receiving an inference request for an input video, partitioning the input video into a plurality of shots, generating, by the trained ensemble of machine learning models, an inference of one or more scene changes in the input video based at least in part on the plurality of shots of the input video, and transmitting the inference to a client application or to a storage location.

Ensemble of machine learning models for automatic scene change detection

Techniques for automatic scene change detection in a video are described. As one example, a computer-implemented method includes extracting features of a query shot and its neighboring shots of a first set of shots without labels with a query model, determining a key shot of the neighboring shots which is most similar to the query shot based at least in part on the features of the query shot and its neighboring shots, extracting features of the key shot with a key model, training the query model into a trained query model based at least in part on a comparison of the features of the query shot and the features of the key shot, extracting features of a second set of shots with labels with the trained query model, and training a temporal model into a trained temporal model based at least in part on the features extracted from the second set of shots and the labels of the second set of shots.

Shot contras five self-supervised learning of a plurality of machine learning models for video analysis applications

Methods and apparatus are described for generating compelling preview clips of media presentations. Compelling clips are identified based on the extent to which human faces are shown and/or the loudness of the audio associated with the clips. One or more of these compelling clips are then provided to a client device for playback.

Leveraging computer vision and machine learning to identify compelling scenes

Techniques for a comic book feature are described herein. A visual data stream of a video may be parsed into a plurality of frames. Scene boundaries may be determined to generate a scene using the plurality of frames where a scene includes a subset of frames. A key frame may be determined for the scene using the subset of frames. An audio portion of an audio data stream of the video may be identified that maps to the subset of frames based on time information. The key frame may be converted to a comic image based on an algorithm. First dimensions and placement for a data object may be determined for the comic image. The data object may include the audio portion for the comic image. A comic panel may be generated for the comic image that incorporates the data object using the determined first dimensions and the placement.

Growing community of inventors

Seattle, WA, United States of America

Shixing Chen