International Business Machines Corporation

Salesforce, Inc.

Massachusetts Institute of Technology

Adobe, Inc.

Logos Care, Inc.

Yang Zhang

Chuang Gan

Dakuo Wang

Shiyu Chang

Michael Reynolds Sollami

Mo Yu

Kaizhi Qian

Sijia Liu

Bo Wu

Tommi S Jaakkola

Ming Tan

Xiaoxiao Guo

Jinjun Xiong

Gautham J Mysore

Yada Zhu

Pin-Yu Chen

David Shing-Ki Kung

Kumar Bhaskaran

Wang Zhou

David A Cox

David E Cox

Subhro Das

Thomas Y Lo

Abhishek Bhandwaldar

Floraine Berthouzoz

Cheng-I Lai

Alexander Haojan Liu

Jason Paul Kovalski

James R Glass

To improve actual labels that are produced by a black box computer classifier system from inputs, identify, using an environment-aware predictor and an environment-agnostic predictor, a subset of the inputs. The subset of the inputs has a stable correlation with the actual labels across a plurality of environments. Identify the subset of the inputs as an explanatory rationale for the actual labels. Display the explanatory rationale with the actual labels to a consumer of the actual labels. Optionally, in response to the explanatory rationale failing a rubric established by the consumer, generate revised inputs by removing the explanatory rationale from the inputs; and produce revised labels by processing the revised inputs with the environment-agnostic predictor.

Game-theoretic invariant rationalization of machine-learning results

Methods, systems, and devices for data processing are described. According to the techniques described herein, a sequential model may be trained using data of different modalities to be used for event recommendation or prediction for an entity or attendee of a future event. Encoders may be used to encode entity data and event data of different data types, and the encoded data may be used to generate vectors for input to a multimodal Transformer. A segment mask may be generated for each of a set of vectors corresponding to the entity and a set of vectors corresponding to an event sequence associated with the entity. The segment masks and sets of vectors may be used to generate embeddings to train the sequential model.

Event prediction based on multimodal learning

Techniques for generator model training are provided. A classifier model that was trained using one or more data samples in a target class is received, and a generative adversarial network (GAN) is trained to generate simulated data samples for the target class, comprising: generating a first simulated data sample using a generator model, computing a first discriminator loss by processing the first simulated data sample using a discriminator model, computing a classifier loss by processing the first simulated data sample using the classifier model, and refining the generator model based on the first discriminator loss and the classifier loss.

Generating data based on pre-trained models using generative adversarial models

One or more computer processors obtain an initial subnetwork at a target sparsity and an initial pruning mask from a pre-trained self-supervised learning (SSL) speech model. The one or more computer processors finetune the initial subnetwork, comprising: the one or more computer processors zero out one or more masked weights in the initial subnetwork specified by the initial pruning mask; the one or more computer processors train a new subnetwork from the zeroed out subnetwork; the one or more computer processors prune one or more weights of lowest magnitude in the new subnetwork regardless of network structure to satisfy the target sparsity. The one or more computer processors classify an audio segment with the finetuned subnetwork.

Self-supervised speech recognition

A vehicle light signal detection and recognition method, system, and computer program product include bounding, using a coarse attention module, one or more regions of an image of an automobile including at least one of a brake light and a signal light generated by automobile signals which include illuminated sections to generate one or more bounded region, removing, using a fine attention module, noise from the one or more bounded regions to generate one or more noise-free bounded regions, and identifying the at least one of the brake light and the signal light from the one or more noise-free bounded regions.

Coarse-to-fine attention networks for light signal detection and recognition

A method (and structure and computer product) for an audiovisual source separation processing, including receiving video data including images of a plurality of sound sources, receiving an optical flow data of the video data, the optical flow data indicating motions of pixels between frames of the video data, and encoding the received video data into video localization data comprising information associating pixels in the frames of video data with different channels of sound.

Audiovisual source separation and localization using generative adversarial networks

Mechanisms are provided for implementing a generative adversarial network (GAN) based restoration system. A first neural network of a generator of the GAN based restoration system is trained to generate an artificial audio spectrogram having a target damage characteristic based on an input audio spectrogram and a target damage vector. An original audio recording spectrogram is input to the trained generator, where the original audio recording spectrogram corresponds to an original audio recording and an input target damage vector. The trained generator processes the original audio recording spectrogram to generate an artificial audio recording spectrogram having a level of damage corresponding to the input target damage vector. A spectrogram inversion module converts the artificial audio recording spectrogram to an artificial audio recording waveform output.

Generative adversarial network based audio restoration

A computer-implemented method is provided of using a machine learning model for disentanglement of prosody in spoken natural language. The method includes encoding, by a computing device, the spoken natural language to produce content code. The method further includes resampling, by the computing device without text transcriptions, the content code to obscure the prosody by applying an unsupervised technique to the machine learning model to generate prosody-obscured content code. The method additionally includes decoding, by the computing device, the prosody-obscured content code to synthesize speech indirectly based upon the content code.

Global prosody style transfer without text transcriptions

A method (and structure and computer product) to permit zero-shot voice conversion with non-parallel data includes receiving source speaker speech data as input data into a content encoder of a style transfer autoencoder system, the content encoder providing a source speaker disentanglement of the source speaker speech data by reducing speaker style information of the input source speech data while retaining content information and receiving target speaker input speech as input data into a target speaker encoder. The output of the content encoder and the target speaker encoder are combined in a decoder of the style transfer autoencoder, and the output of the decoder provides the content information of the input source speech data in a style of the target speaker speech information.

High-quality non-parallel many-to-many voice conversion

Computer implemented method and a system for auto completion of text based on the context associated with the text. The computer implemented method includes steps of receiving input text, identifying a certain context associated with the input text from multiple predefined contexts, by feeding the input text into a context-prediction component of a machine learning model that predicts the certain context, selecting a certain context-specific component of the machine learning model from multiple context-specific components according to the identified certain context, feeding the input text into the selected context-specific component that outputs autocomplete text associated with the identified certain context. The context-specific components are each trained to generate autocompleted text associated with a respective context pre-defined for the respective context-specific component.

Autocomplete of user entered text

Systems and techniques that facilitate few-shot temporal action localization based on graph convolutional networks are provided. In one or more embodiments, a graph component can generate a graph that models a support set of temporal action classifications. Nodes of the graph can correspond to respective temporal action classifications in the support set. Edges of the graph can correspond to similarities between the respective temporal action classifications. In various embodiments, a convolution component can perform a convolution on the graph, such that the nodes of the graph output respective matching scores indicating levels of match between the respective temporal action classifications and an action to be classified. In various embodiments, an instantiation component can input into the nodes respective input vectors based on a proposed feature vector representing the action to be classified. In various cases, the respective temporal action classifications can correspond to respective example feature vectors, and the respective input vectors can be concatenations of the respective example feature vectors and the proposed feature vector.

Framework for few-shot temporal action localization

Systems and methods are provided for receiving an image that includes a clothed person, determining a pose of the person in the image, and segmenting the image into one or more first fashion items. One or more second fashion items may be determined using a similarity search that searches at least one storage device communicatively coupled to the server based on the one or more first fashion items. At least one outfit proposal may be generated based on the one or more second fashion items. Image re-stylization of corresponding portions of the image may be performed, including the clothed person to generate recommended outfit images based on the at least one outfit proposal. The generated outfit images may be transmitted for display.

Systems and methods of image-based neural network apparel recommendation

Dual-modality relation networks for audio-visual event localization can be provided. A video feed for audio-visual event localization can be received. Based on a combination of extracted audio features and video features of the video feed, informative features and regions in the video feed can be determined by running a first neural network. Based on the informative features and regions in the video feed determined by the first neural network, relation-aware video features can be determined by running a second neural network. Based on the informative features and regions in the video feed, relation-aware audio features can be determined by running a third neural network. A dual-modality representation can be obtained based on the relation-aware video features and the relation-aware audio features by running a fourth neural network. The dual-modality representation can be input to a classifier to identity an audio-visual event in the video feed.

Dual-modality relation networks for audio-visual event localization

Systems and methods are provided for receiving at least a first query color, and searching an electronic catalog including a plurality of product images for the first query color to determine a similarity measure between the first query color and a product image of a plurality of product images. The similarity measure may be determined by determining a Euclidean distance between values in a three-dimensional color space for the first query color and a target color of the product image, and determining the similarity measure between the query color and the product image by determining a sum of the similarity measures from all target colors on the product image, weighted by the coverage of each target color. The search results may be transmitted based on the searching of the electronic catalog including the plurality of product images for the first query color.

Growing community of inventors

Cambridge, MA, United States of America

Yang Zhang