System for replacing elements in content

Abstract

A system for contextual modification of content based on multimodal extraction of metadata from the content. The metadata is extracted by processing one or more scenes in the content to extract metadata corresponding to multiple extraction modes, and an embedding model for each extraction mode wherein an aggregated embedding model responsive to the extracted metadata for each mode formulates an aggregated embedding. A process controller may include an embedding extractor responsive to a control input. The control input may specify one or more features appearing in the content defining a content modification opportunity. The embedding extractor may include an embedding model coordinated with the embedding model for one or more of the embedding modes to generate an opportunity embedding in the form of a vector. A vector comparison processor determines the distance between the opportunity embedding and the aggregated embedding, wherein the embeddings are in the form of vectors. The process controller is responsive to the vector comparison processor to generate edit control instructions indicating a modification of the content upon detection of the content modification opportunity. A preference database may contain content modification preferences for a subset of consumers. The process controller may be connected to the preference database to obtain content modification preferences for the consumers and the control instructions may include identification of the consumers and the content modification preferences for the consumers are connected to the embedding extractor. A content editor is responsive to the edit control instructions to modify the content. The content editor uses generative AI techniques to modify the content by replacing an element appearing within the content during the content modification opportunity with an element correlated with the element appearing in said content.