Meta Platforms, Inc.

Facebook, Inc.

Lorenzo Torresani

Heng Wang

Du Le Hong Tran

Gediminas Bertasius

Kaiming He

Matthew Dan Feiszli

Devi Niru Parikh

Jue Wang

Antoine Miech

Xudong Lin

A system and method for generating task-specific text by processing multimodal inputs using machine-learning models is provided. The method may include accessing first sets of tokens associated with a desired task and one or more modalities associated with a context of the desired task. The method may further include determining a second set of tokens for each of the one or more modalities using a classifier network associated with the modality. The method may further include generating a number of embedding vectors by mapping the first sets of tokens and the second set of tokens associated with each of the one or more modalities to an embedding space. The method may further include producing a sequence of words addressing the desired task by processing the number of embedding vectors with an encoder-decoder network.

Task-specific text generation based on multimodal inputs

In one embodiment, a method includes accessing a stream of F video frames, where each of the F video frames includes N patches that are non-overlapping, generating an initial embedding vector for each of the N×F patches in the F video frames, generating a classification embedding by processing the generated N×F initial embedding vectors using a self-attention-based machine-learning model that computes a temporal attention and a spatial attention for each of the N×F patches, and determining a class of the stream of video frames based on the generated classification embedding.

Classifying a video stream using a self-attention-based machine-learning model

In one embodiment, a method includes accessing a first set of images of multiple images of a scene, wherein the first set of images show the scene during a time period. The method includes generating, by processing the first set of images using a first machine-learning model, one or more attributes representing observed actions performed in the scene during the time period. The method includes predicting, by processing the generated one or more attributes using a second machine-learning model, one or more actions that would happen in the scene after the time period.

Anticipating future video based on present video

In one embodiment, a method includes receiving a request for information associated with a video, determining the information associated with the video by processing the video using a machine-learning model which is based on a convolutional neural network comprising a plurality of layers, wherein at least one of the plurality of layers comprises one or more building blocks, wherein at least one of the one or more building blocks comprises a first filter configured to perform a three-dimensional (3D) pointwise convolutional operation and a second filter configured to perform a three-dimensional (3D) groupwise convolutional operation, and outputting the information associated with the video in response to the request.

Convolutional neural network based on groupwise convolution for efficient video analysis

Growing community of inventors

Palo Alto, CA, United States of America

Lorenzo Torresani