Meta Platforms, Inc.

Gediminas Bertasius

Lorenzo Torresani

Heng Wang

Devi Niru Parikh

Jue Wang

Xudong Lin

A system and method for generating task-specific text by processing multimodal inputs using machine-learning models is provided. The method may include accessing first sets of tokens associated with a desired task and one or more modalities associated with a context of the desired task. The method may further include determining a second set of tokens for each of the one or more modalities using a classifier network associated with the modality. The method may further include generating a number of embedding vectors by mapping the first sets of tokens and the second set of tokens associated with each of the one or more modalities to an embedding space. The method may further include producing a sequence of words addressing the desired task by processing the number of embedding vectors with an encoder-decoder network.

Task-specific text generation based on multimodal inputs

In one embodiment, a method includes accessing a stream of F video frames, where each of the F video frames includes N patches that are non-overlapping, generating an initial embedding vector for each of the N×F patches in the F video frames, generating a classification embedding by processing the generated N×F initial embedding vectors using a self-attention-based machine-learning model that computes a temporal attention and a spatial attention for each of the N×F patches, and determining a class of the stream of video frames based on the generated classification embedding.

Growing community of inventors

Boston, MA, United States of America

Gediminas Bertasius