Facebook, Inc.

Waveone Inc.

Apple Inc.

Meta Platforms, Inc.

Lubomir Dimitrov Bourdev

Oren Rippel

Balamanohar Paluri

Sanjay Nair

Robert D Fergus

Carissa Lew

Alexander Anderson

Apostolos Lerios

Dirk John Stoop

Balmanohar Paluri

Ryan David Mack

Hervé Guihot

Kedar Tatwawadi

Brandon Sprague

Craig S Lytle

Du Le Hong Tran

Kevin Dechau Tang

Sumit Chopra

Yaniv Taigman

Nikhil Johri

Ning Zhang

Steve Branson

Fei Yang

Sainbayar Sukhbaatar

Louis Dominic Brandy

Steven Branson

Hassan Almas

Yunchao Gong

Marcin Pawlowski

Alexander G Anderson

Craig Lytle

An autoencoder is configured to encode content at different quality levels. The autoencoder includes an encoding system and a decoding system with neural network layers forming an encoder network and a decoder network. The encoder network and decoder network are configured to include branching paths through the networks that include different subnetworks. During deployment, content is provided to the encoding system with a quality signal indicating a quality at which the content can be reconstructed. The quality signal determines which of the paths through the encoder network are activated for encoding the content into one or more tensors, which are compressed into a bitstream and later used by the decoding system to reconstruct the content. The autoencoder is trained by randomly or systematically selecting different combinations of tensors to use to encode content and backpropagating error values from loss functions through the network paths associated with the selected tensors.

Dynamic control for a machine learning autoencoder

A compression system trains a machine-learned compression model that includes components for an encoder and decoder. In one embodiment, the compression model is trained to receive parameter information on how a target frame should be encoded with respect to one or more encoding parameters, and encodes the target frame according to the respective values of the encoding parameters for the target frame. In particular, the encoder of the compression model includes at least an encoding system configured to encode a target frame and generate compressed code that can be transmitted by, for example, a sender system to a receiver system. The decoder of the compression model includes a decoding system trained in conjunction with the encoding system. The decoding system is configured to receive the compressed code for the target frame and reconstruct the target frame for the receiver system.

Parameter map for machine-learned video compression

A cloud service system manages a filter repository including filters for encoding and decoding media content (e.g. text, image, audio, video, etc.). The cloud service system may receive a request from a client device to provide a filter for installation on a node such as an endpoint device (e.g. pipeline node). The request includes information such as a type of bitstream to be processed by the requested filter. The request may further include other information such as hardware configuration and functionality attribute. The cloud service system may access the filter repository that stores the plurality of filters including encoder filters and decoder filters and may select a filter that is configured to process the type of bitstream identified in the request and provide the selected filter to the client device.

System for training and deploying filters for encoding and decoding

A deep learning based compression (DLBC) system trains multiple models that, when deployed, generates a compressed binary encoding of an input image that achieves a reconstruction quality and a target compression ratio. The applied models effectively identifies structures of an input image, quantizes the input image to a target bit precision, and compresses the binary code of the input image via adaptive arithmetic coding to a target codelength. During training, the DLBC system reconstructs the input image from the compressed binary encoding and determines the loss in quality from the encoding process. Thus, the models can be continually trained to, when applied to an input image, minimize the loss in reconstruction quality that arises due to the encoding process while also achieving the target compression ratio.

Deep learning based on image encoding and decoding

A compression system trains a compression model for an encoder and decoder. In one embodiment, the compression model includes a machine-learned in-loop flow predictor that generates a flow prediction from previously reconstructed frames. The machine-learned flow predictor is coupled to receive a set of previously reconstructed frames and output a flow prediction for a target frame that is an estimation of the flow for the target frame. In particular, since the flow prediction can be generated by the decoder using the set of previously reconstructed frames, the encoder may transmit a flow delta that indicates a difference between the flow prediction and the actual flow for the target frame, instead of transmitting the flow itself. In this manner, the encoder can transmit a significantly smaller number of bits to the receiver, improving computational efficiency.

Machine-learned in-loop predictor for video compression

A deep learning based compression (DLBC) system applies trained models to compress binary code of an input image to a target codelength. For a set of binary codes representing the quantized coefficents of an input image, the DLBC system applies a first model that is trained to predict feature probabilities based on the context of each bit of the binary codes. The DLBC system compresses the binary code via adaptive arithmetic coding based on the determined probability of each bit. The compressed binary code represents a balance between a reconstruction quality of a reconstruction of the input image and a target compression ratio of the compressed binary code.

Deep learning based adaptive arithmetic coding and codelength regularization

In one embodiment, a method includes locating a plurality of part patches from an image, wherein each part patch comprises at least a portion of the image corresponding to a recognized human body portion or pose, and wherein each part patch is associated with a respective detection score larger than a threshold score, wherein the detection score is determined based on a comparison between the part patch with multiple training patches, generating a plurality of sets of feature data by processing each of the plurality of part patches with a plurality of convolutional neural networks, respectively, and determining whether a human attribute exists in the image based on the plurality of sets of feature data.

Pose-aligned networks for deep attribute modeling

The compression system trains a machine-learned encoder and decoder through an autoencoder architecture. The encoder can be deployed by a sender system to encode content for transmission to a receiver system, and the decoder can be deployed by the receiver system to decode the encoded content and reconstruct the original content. The encoder is coupled to receive content and output a tensor as a compact representation of the content. The content may be, for example, images, videos, or text. The decoder is coupled to receive a tensor representing content and output a reconstructed version of the content. The compression system trains the autoencoder with a discriminator to reduce compression artifacts in the reconstructed content. The discriminator is coupled to receive one or more input content, and output a discrimination prediction that discriminates whether the input content is the original or reconstructed version of the content.

Using generative adversarial networks in compression

A machine learning (ML) task system trains a neural network model that learns a compressed representation of acquired data and performs a ML task using the compressed representation. The neural network model is trained to generate a compressed representation that balances the objectives of achieving a target codelength and achieving a high accuracy of the output of the performed ML task. During deployment, an encoder portion and a task portion of the neural network model are separately deployed. A first system acquires data, applies the encoder portion to generate a compressed representation, performs an encoding process to generate compressed codes, and transmits the compressed codes. A second system regenerates the compressed representation from the compressed codes and applies the task model to determine the output of a ML task.

Data compression for machine learning tasks

Systems, methods, and non-transitory computer-readable media can obtain a first batch of content items to be clustered. A set of clusters can be generated by clustering respective binary hash codes for each content item in the first batch, wherein content items included in a cluster are visually similar to one another. A next batch of content items to be clustered can be obtained. One or more respective binary hash codes for the content items in the next batch can be assigned to a cluster in the set of clusters.

Systems and methods for online clustering of content items

A deep learning based compression (DLBC) system generates a progressive representation of the encoded input image such that a client device that requires the encoded input image at a particular target bitrate can readily be transmitted the appropriately encoded data. More specifically, the DLBC system computes a representation that includes channels and bitplanes that are ordered based on importance. For a given target rate, the DLBC system truncates the representation according to a trained zero mask to generate the progressive representation. Transmitting a first portion of the progressive representation enables a client device with the lowest target bitrate to appropriately playback the content. Each subsequent portion of the progressive representation allows the client device to playback the content with improved quality.

Enhanced coding efficiency with progressive representation

Growing community of inventors

Mountain View, CA, United States of America

Lubomir Dimitrov Bourdev