Amazon Technologies, Inc.

Naveen Sudhakaran Nair

Pragyana K Mishra

Wenbin Ouyang

Mohammed Khalilia

Rohun Tripathi

Angshuman Saha

Rajeev Ramnarain Rastogi

Gregory Johnson

Pooja Ashok Kumar

Ali Abdool

Baris Gecer

Anton Ivchenko

Chittaranjan Tripathy

Systems and techniques for validation and generation of localized content for audio and video are described herein. The systems and techniques provide for training of twin neural networks to evaluate performance characteristics, sometimes referred to as content-auxiliary characteristics, of a localized performance. The localized performance may be validated or improved by identifying misalignment in the performance characteristics to ensure that localized content preserves content as well as creative intent and performance ability in the final product. The machine learning models trained using the techniques described herein may be used in connection with auto-localization processes to automatically generate high quality localized audio and video content.

Performance characteristic transfer for localized content

Systems and methods to generate synthetic lip synchronization may generate source facial keypoints based on a source video, generate target facial keypoints based on a target audio, determine distances between the source facial keypoints and target facial keypoints, and transform or warp the source facial keypoints and associated surfaces to the target facial keypoints. In this manner, target video having synthetic lip synchronization that matches the target audio may be generated, and the target video may substantially preserve or maintain surface textures or features from the source video in the target video, thereby generating natural and believable synthetic lip synchronization corresponding to the target audio.

Systems and methods to generate synthetic lip synchronization having faithful textures

A computer-implemented method is disclosed. The method includes selecting one or more target surfaces portrayed in at least one video frame, generating a video data latent space representation of the at least one video frame, accessing a plurality of supplemental data latent space representations of a plurality of supplemental data sets, identifying a particular supplemental data latent space representation based at least in part on the video data latent space representation, selecting a particular supplemental data set in response to identifying the particular supplemental data latent space representation, the particular supplemental data set corresponding with the particular supplemental data latent space representation, and inserting the particular supplemental data set into the at least one video frame.

Inserting supplemental data into image data

Systems, devices, and methods are provided for training and/or inferencing using machine-learning models. In at least one embodiment, a user selects a source media (e.g., video or audio file) and a target identity. A content embedding may be extracted from the source media, and an identity embedding may be obtained for the target identity. The content embedding of the source media and the identity embedding of the target identity may be provided to a transfer model that generates synthesized media. For example, a user may select a song that is sung by a first artist and then select a second artist as the target identity to produce a cover of the song in the voice of the second artist.

Identity transfer models for generating audio/video content

Devices, systems, and methods are provided for presenting on-screen text during video playback. A method may include detecting a user request to determine when to activate and deactivate presentation of on-screen text during playback of a video; inputting, to a machine learning model, text data of video titles, audio data of the video titles, video frames of the video titles, and user data associated with users of a streaming video application; generating, using the machine learning model, based on the text data, the audio data, the video frames, and the user data, the first times and the second times; sending a bitstream comprising streaming video and indications of the first times and the second times; activating, based on the first times, presentation of the on-screen text during presentation of the streaming video; and deactivating, based on the second times, presentation of the on-screen text during presentation of the streaming video.

Enhanced control of video subtitles

Systems, devices, and methods are provided for multi-stem volume equalization, wherein the volume levels of each stem may be adjusted non-uniformly. Audio may be diarized into a plurality of stems, including background noise separate. Mean and variance of the volume levels of the stems may be computed. Each audio stem may be automatically adjusted based on a stem-specific preference that a user may specify. View may adjust actor volume relative to the mean/variance that maintains a relative difference in volume levels between stems.

Multi-channel volume level equalization based on user preferences

Techniques are generally described for causal impact estimation using machine learning. A first machine learning model is trained using non-treatment variables during training. A second machine learning model uses learned weights from the first machine learning model for non-treatment variables and is trained on one or more treatment variables. The second machine learning model estimates outcomes based on the presence or absence of an event represented by the treatment variable. Selection bias is reduced by warm-starting the second machine learning model with non-treatment variable weights learned during training of the first machine learning model.

Causal impact estimation model using warm starting for selection bias reduction

Prediction of outcomes of disruptive treatments are enabled utilizing sequenced training of a machine learning model over ordered bins of treatment candidates. Treatment candidates may be assigned to candidate characterization bins with an ordering, and the model may be trained with a sequence of training steps corresponding to the ordering of the candidate characterization bins, in each training step the model having untreated candidate features from a corresponding bin and aggregate metrics from one or more previous steps as input. The predicted outcome for a selected bin may be generated with the trained model having treated candidate features and aggregate metrics from one or more previous steps as input. The predicted outcome may be a counterfactual prediction for a bin with insufficient control candidates, and may represent a nonlinear extrapolation from control data in prior bins in the bin ordering.

Disruptive prediction with ordered treatment candidate bins

Model-based dubbing techniques are implemented to generate a translated version of a source video. Spoken audio portions of a source video may be extracted and semantic graphs generated that represent the spoken audio portions. The semantic graphs may be used to produce translations of the spoken portions. A machine learning model may be implemented to generate replacement audio for the spoken portions using the translation of the spoken portion. A machine learning model may be implemented to generate modifications to facial image data for a speaker of the replacement audio.

Model-based dubbing to translate spoken audio in a video

Audio content and played frames may be received. The audio content may correspond to first video content. The played frames may be included in the first video content. The first video content may further include a replaced frame. The played frames and the replaced frame may include a face of a person. Location data may also be received that indicates locations of facial features of the face of the person within the replaced frame. A replacement frame may be generated, such as by rendering the facial features in the replacement frame based at least in part on the locations indicated by the location data and positions indicated by a portion of the audio content that is associated with the replaced frame. Second video content may be played including the played frames and the replacement frame. The replacement frame may replace the replaced frame in the second video content.

Video frame replacement based on auxiliary data

An approximate count of a subset of records of a data set is obtained using one or more transformation functions. The subset comprises records which contain a first value of one input variable, a second value of another input variable, and a particular value of a target variable. Using the approximate count, an approximate correlation metric for a multidimensional feature and the target variable is obtained. Based on the correlation metric, the multidimensional feature is included in a candidate feature set to be used to train a machine learning model.

Scalable generation of multidimensional features for machine learning

Systems and methods are provided for using a machine learning model to analyze customers data for each of a number of customers interacting with electronic commerce, and determining whether the customers are high value customers. For the high value customers, one or more high value actions are then determined by the machine learning model, the one or more high value actions representing actions previously performed by or with respect to the customers and that correlate with a likelihood of the customers reaching high value customers status. Non-high value customers may then be prompted to perform the high value action which may elevate the non-high value customers to high value customers status.

High value action selection using a machine learning model

Techniques for generating an automated guide for exploring content of a content streaming platform based on inferred intent are described herein. For example, click stream data from interactions with a content streaming platform during a session, along with historical click stream data of historical interactions with the content streaming platform during historical sessions may be obtained. A score for each piece of content from a plurality of content offered by the content streaming platform may be determined based on the click stream data and the historical click stream data. The plurality of content may be ranked based on the associated scores for each piece of content. Media content may be generated that includes a portion of an associated trailer from a subset of the ranked plurality of content based on the score for each piece of content of the subset. The media content may be presented via a user interface.

Automated and guided video content exploration and discovery

A content provider may cause a client device of a user to output a personalized puzzle in response to receiving a request from the client device to access electronic content of the content provider. The puzzle may include a theme that corresponds to a determined predilection of the user, and/or the puzzle may be a type of puzzle that corresponds to the user's predilection. The client device may also output, with the puzzle, an incentive for completing (e.g., solving) the puzzle Upon receiving data indicating that the user has completed his/her personalized puzzle, the content provider may provide the reward to the user.

Growing community of inventors

Issaquah, WA, United States of America

Naveen Sudhakaran Nair