Samsung Electronics Co., Ltd.

International Business Machines Corporation

Hongxia Jin

Yilin Shen

Jeffrey Bruce Lotspiech

Avik Ray

Qihua Wang

Burak Uzkent

Sigfredo Ismael Nin

Wenbo Li

Yu Wang

Yue Deng

Robert Glenn Deen

Ginger Marie Myles

Shiva Prasad Kasiviswanathan

Sandeep Nama

Dalkandura Arachchige Kalpa Shashika Silva Gunaratna

Christian Koehler

Vasili Ramanishka

Alan Messer

Scott Schumacher

Richard Chow

Yan Liu

Na Wang

Rui Chen

Justin C Martineau

Gokay Saldamli

Donald E Leake, Jr

Qian Lou

Yen-Chang Hsu

Wilfred E Plouffe

Xiangyu Zeng

Philip Anastasios Zigoris

Wenjun Zhou

Bart Knijnenburg

Pengfei Hu

Deguang Kong

Nimrod Megiddo

Dulce Beatriz Ponceleon

Larry Paul Heck

Thomas Alexander Bellwood

Matthew Francis Rutkowski

Jeffrey Scott Pierce

Hernan Badenes

Avinash Laxmisha Varna

Bin Liu

Julian Ariel Cerruti

Wei Yang

Mateo Nicolas Bengualid

Dawei Li

Tong Yu

Serdar Pehlivanoglu

Rakshith Sharma Srinivasa

Changsheng Zhao

Ting Hua

Chou-Chang Yang

Ching-Hua Lee

Haoran Li

Yashas Malur Saidutta

Peter M Bretan

Shangqian Gao

Yi Wei

Zhipeng Mo

Jing Zhu

Manas Gaur

Peixi Xiong

Fenjiao Wang

Ruoxu Wang

Nina Narodytska

Di Ma

Jeffrey Bruce Lotsprech

Michael Joseph Nelson

Jeffrey Bruce Lotsplech

A method includes obtaining raw image data, where the raw image data includes data values each having most significant bits and least significant bits. The method also includes providing the raw image data to a trained machine learning model and generating processed image data using the trained machine learning model. The method further includes presenting an image based on the processed image data. The trained machine learning model is trained to modulate a feature map associated with the most significant bits of the data values of the raw image data based on the least significant bits of the data values of the raw image data in order to generate a fusion of the most significant bits and the least significant bits of the data values of the raw image data.

Fusion techniques for combining most significant bits and least significant bits of image data in image processing or other applications

A method of a server device is provided. The method of a server device includes retrieving a prediction input and a prediction setting, replacing at least one non-linear activation channel in a neural network with at least one replacement channel based on the received prediction setting, generating a prediction based on the received prediction input based on the neural network with the at least one replacement channel, and outputting the generated prediction.

Secure, accurate and fast neural network inference by replacing at least one non-linear activation channel

A method of training a model includes configuring a first transformer for visual learning with a first set of weights, configuring a second transformer for textual learning with a second set of weights, adjusting at least the second set of weights based on minimizing a weight difference between the first set of weights and the second set of weights, replacing the first set of weights for the first transformer with the adjusted second set of weights, and updating the first transformer based on the adjusted second set of weights.

Method and system for learning to share weights across transformer backbones in vision and language tasks

A continual learning method includes obtaining an input data including a trained model, continual learning (CL) Information, and training data by an electronic device. The method also includes re-training, using the electronic device, the model for a task based on the training data. The method also includes updating, using the electronic device, the CL Information based on the model and the training data. The method further includes selecting a first set of exemplars from the training data based on data associated with the CL Information. The CL Information includes a first group of variables associated with the model and a second group of variables associated with the model that changes to the first group of variables have stronger impact to the model's performance of the task than changes to the second group of variables.

Systems and methods for continual learning

Computer-based generation of information-gathering questions in response to a user query can include parsing the user query using natural language processing and extracting from the user query one or more phrases corresponding to a predetermined category. A knowledge database can be accessed, and entities semantically related to each phrase can be extracted therefrom. A query sub-graph representing a relationship between each of the one or more phrases and the entities extracted from the knowledge database can be generated. An expanded user query can be generated by traversing the query sub-graph. Passages semantically related to the expanded user query can be retrieved from one or more passages databases and ranked. A neural question generator can generate a set of information-gathering questions based on the expanded user query and a group of select passages selected from the plurality of passages in accordance with each selected passage's ranking.

Dynamic question generation for information-gathering

An apparatus for performing image processing, may include at least one processor configured to: input an image to a vision transformer comprising a plurality of encoders that correspond to at least one fixed encoder and a plurality of adaptive encoders; process the image via the at least one fixed encoder to obtain image representations; determine one or more layers of the plurality of adaptive encoders to drop, by inputting the image representations to a policy network configured to determine layer dropout actions for the plurality of adaptive encoders; and obtain a class of the input image using remaining layers of the plurality of adaptive encoders other than the dropped one or more layers.

Method and apparatus for classifying images using an artificial intelligence model

A method of performing a multimodal tasks by using a multimodal model that includes a text encoder and a vision encoder, may include obtaining a text feature from the query via the text encoder; obtaining an image feature from the one or more input images via the vision encoder; and outputting a response to the query based on similarity between the text feature and the image feature, wherein weights vectors of the text encoder and the vision encoder are pruned and shared according to a sharing vector and a pruning vector that are generated by a hypernetwork, and wherein the hypernetwork and the multimodal model are jointly trained to minimize at least one of a difference between the weight vectors in the text encoder and the vision encoder, a difference between the weight vectors in different layers of the text encoder, and a number of parameters in the multimodal model.

Apparatus and method for sharing and pruning weights for vision and language models

A method includes obtaining, using a first electronic device, a weight matrix associated with a trained transformer model. The method also includes factorizing the weight matrix into a dictionary weight matrix and an intermediate matrix. The method further includes pruning the intermediate matrix to generate a sparse intermediate matrix. The method also includes fine-tuning the sparse intermediate matrix based on a training dataset to generate a fine-tuned sparse intermediate matrix. The method further includes determining an index matrix and a coefficient matrix based on the fine-tuned sparse intermediate matrix. In addition, the method includes deploying the dictionary weight matrix, the index matrix, and the coefficient matrix to a second electronic device without deploying the weight matrix to the second electronic device. A number of parameters in the dictionary weight matrix, the index matrix, and the coefficient matrix is smaller than a number of parameters in the weight matrix.

Small and fast transformer model for multi-modal or other tasks

A method of training a neural network model includes generating a positive image based on an original image, generating a positive text corresponding to the positive image based on an original text corresponding to the original image, the positive text referring to an object in the positive image, constructing a positive image-text pair for the object based on the positive image and the positive text, constructing a negative image-text pair for the object based on the original image and a negative text, the negative text not referring to the object, training the neural network model based on the positive image-text pair and the negative image-text pair to output features representing an input image-text pair, and identifying the object in the original image based on the features representing the input image-text pair.

Supervised contrastive learning for visual grounding

Computer-based content understanding can include segmenting an image into a plurality of blocks, wherein each block includes textual information from the image. For each block of the plurality of blocks, encoded feature data is generated by encoding visual information of the block and visual information of one or more neighboring blocks from the plurality of blocks and encoded textual data is generated by encoding the textual information of the block and the textual information of the one or more neighboring blocks. Further, using an entity class prediction model, one or more tokens of the block are classified into one or more entity classes based on a combination of the encoded textual data and the encoded feature data. A plurality of entities can be extracted from the image based on the entity classes of the plurality of blocks.

On-device knowledge extraction from visually rich documents

A method includes obtaining, using at least one processing device, noisy speech signals and extracting, using the at least one processing device, acoustic features from the noisy speech signals. The method also includes receiving, using the at least one processing device, a predicted speech mask from a speech mask prediction model based on a first acoustic feature subset and receiving, using the at least one processing device, a predicted noise mask from a noise mask prediction model based on a second acoustic feature subset. The method further includes providing, using the at least one processing device, predicted speech features determined using the predicted speech mask and predicted noise features determined using the predicted noise mask to a filtering mask prediction model. In addition, the method includes generating, using the at least one processing device, a clean speech signal using a predicted filtering mask output by the filtering mask prediction model.

Speech denoising networks using speech and noise modeling

A method includes identifying multiple tokens contained in an input utterance. The method also includes generating slot labels for at least some of the tokens contained in the input utterance using a trained machine learning model. The method further includes determining at least one action to be performed in response to the input utterance based on at least one of the slot labels. The trained machine learning model is trained to use attention distributions generated such that (i) the attention distributions associated with tokens having dissimilar slot labels are forced to be different and (ii) the attention distribution associated with each token is forced to not focus primarily on that token itself.

Apparatus and method for compositional spoken language understanding

In one embodiment, a method includes accessing an image and a natural-language question regarding the image and extracting, from the image, a first set of image features at a first level of granularity and a second set of image features at a second level of granularity. The method further includes extracting, from the question, a first set of text features at the first level of granularity and a second set of text features at the second level of granularity; generating a first output representing an alignment between the first set of image features and the first set of text features; generating a second output representing an alignment between the second set of image features and the second set of text features; and determining an answer to the question based on the first output and the second output.

Multi-granularity alignment for visual question answering

A method includes obtaining a batch of training data including multiple paired image-text pairs and multiple unpaired image-text pairs, where each paired image-text pair and each unpaired image-text pair includes an image and a text. The method also includes training a machine learning model using the training data based on an optimization of a combination of losses. The losses include, for each paired image-text pair, (i) a first multi-modal representation loss based on the paired image-text pair and (ii) a second multi-modal representation loss based on two or more unpaired image-text pairs, selected from among the multiple unpaired image-text pairs, wherein each of the two or more unpaired image-text pairs includes either the image or the text of the paired image-text pair.

System and method for supervised contrastive learning for multi-modal tasks

An inpainting method includes obtaining an image including an object having a delicate shape and identifying a target region within the image, where the target region is adjacent to the object. The method also includes using a first mask to separate the image into a number of semantic categories and aggregating neighboring contexts for the target region based on the semantic categories. The method further includes restoring, based on the aggregated contexts, textures in the target region without affecting the delicate shape of the object. In addition, the method includes displaying a refined image including the restored textures in the target region and the object.

System and method for image inpainting based on layout-guided pre-processing attention mechanism

Growing community of inventors

San Jose, CA, United States of America

Hongxia Jin