Microsoft Technology Licensing, LLC

Robert Boschgmbh

Mohsen Fayyaz

Eric Chris Wolfgang Sommerlade

Hamidreza Vaezi Joze

Justin James Wagle

Nazuk Jain

Nadine Behrmann

Mehdi Noroozi

Soroush Abbasi Koohpayegani

A technique uses an encoder system to produce an index of target item embeddings. Each target item embedding is input-agnostic and universal in the sense that different expressions of a target concept, produced using different combinations of input modes, map to the same target item embedding in the index. The encoder system throttles the amount of computations it performs based on the assessed capabilities of an execution platform. A retrieval system processes a multimodal input query by first generating a candidate set of target item embeddings in the index that match the input query, and then using a filtering operation to identify those target item embeddings that are most likely to match the input query. The encoder system and the retrieval system rely on language-based components having weights that are held constant during a training operation. Other weights of these systems are updated during the training operation.

Using fixed-weight language models to create and interact with a retrieval index

A technique executes tasks using a data store of machine-trained models. The data store specifically includes a subset of encoder-type machine-trained models for converting input data items having different input data types into respective embeddings in a vector space, and a subset of decoder-type machine-trained models for converting embeddings in the same vector space into data items having respective different output data types. When executing a particular task that involves one or more data types, the technique selects one or more machine-trained models that match those data types. In some implementations, the technique provides a clipboard store for storing embeddings produced by the encoder-type machine-trained models and consumable by the decoder-type machine-trained models. The technique includes provisions for ensuring that any decoder-type machine-model is capable of processing embeddings produced by different versions of the encoder-type machine-trained models.

Performing computing tasks using decoupled models for different data types

A method for coding a predefined time sequence of video images in a representation which is evaluable by machine made up of stationary features and nonstationary features. In the method: at least one function parameterized using trainable parameters is provided, which maps sequences of video images on representations; from the sequence of video images, N adjoining, nonoverlapping short extracts and one long extract, which contains all N short extracts are selected; using the parameterized function, a representation of the long extract and multiple representations of the short extracts are ascertained; the parameterized function is assessed; the parameters of the function are optimized with the goal that the assessment of the cost function for representations ascertained in future is expected to improve; using the function parameterized by the finished optimized parameters, the predefined time sequence of video images is mapped on the sought representation.

Method for coding a sequence of video images

A transformer is described herein for using transformer-based technology to process data items (e.g., image items). The transformer increases the efficiency of the transformer-based technology by using a modified attention component. In operation, the modified attention component accepts embedding vectors that represent a plurality of item tokens, together with a classification token. A first stage of the modified attention component generates original attention information based on the embedding vectors. A second stage generates score information based on a portion of the original attention information that pertains to the classification token. A third stage produces modified attention information by removing attention values from the original attention information, as guided by a sampling operation that is performed on the score information. The second and third stages do not rely on machine-trained values, which expedites the deployment of these functions in existing transformers.

Adaptive token sampling for efficient transformer

A technique is described herein that interprets some frames in a stream of video content as key frames and other frames as predicted frames. The technique uses an image analysis system to produce feature information for each key frame. The technique uses a prediction model to produce feature information for each predicted frame. The prediction model operates on two inputs: (1) feature information that has been computed for an immediately-preceding frame; and (2) frame-change information. A motion-determining model produces the frame-change information by computing the change in video content between the current frame being predicted and the immediately-preceding frame. The technique reduces the amount of image-processing operations that are used to process the stream of video content compared to a base case of processing all of the frames using the image analysis system. As such, the technique uses less computing resources compared to the base case.

Growing community of inventors

Berlin, Germany

Mohsen Fayyaz