Meta Platforms, Inc.

Xudong Lin

Lorenzo Torresani

Gediminas Bertasius

Jue Wang

Devi Niru Parikh

A system and method for generating task-specific text by processing multimodal inputs using machine-learning models is provided. The method may include accessing first sets of tokens associated with a desired task and one or more modalities associated with a context of the desired task. The method may further include determining a second set of tokens for each of the one or more modalities using a classifier network associated with the modality. The method may further include generating a number of embedding vectors by mapping the first sets of tokens and the second set of tokens associated with each of the one or more modalities to an embedding space. The method may further include producing a sequence of words addressing the desired task by processing the number of embedding vectors with an encoder-decoder network.

Growing community of inventors

New York, NY, United States of America

Xudong Lin