Salesforce Com, Inc.

Salesforce, Inc.

Shafiq Rayhan Joty

Chu Hong Hoi

Yue Wang

Weishi Wang

Samson Min Rong Tan

Amrita Saha

Junnan Li

Chien-Sheng Wu

Akhilesh Deepak Gotmare

Yifan Gao

Xuan Phi Nguyen

Guangsen Wang

Ka Chun Au

Hailin Chen

Gang Wu

Embodiments described herein provide systems and methods for a partially supervised training model for questioning answering tasks. Specifically, the partially supervised training model may include two modules—a query parsing module and a program execution module. The query parsing module parses queries into a grogram, and the program execution module execute the program to reach an answer through explicit reasoning and partial supervision. In this way, the partially supervised training model can be trained with answers as supervision, obviating the need for supervision by gold program operations and gold query-span attention at each step of the program.

Systems and methods for numerical reasoning by a partially supervised numeric reasoning module network

Embodiments described herein provides a contrastive learning framework that leverages hard negative examples, that are mined globally from the entire training corpus for a given query to improve the quality of code and natural language representations. Specifically, similar examples from the training corpus are extracted and used as hard negatives in an online manner during training while keeping the minibatch construction random.

Systems and methods for semantic code search

Embodiments described herein a code generation and understanding model that builds on a Transformer-based encoder-decoder framework. The code generation and understanding model is configured to derive generic representations for programming language (PL) and natural language (NL) in code domain via pre-training on unlabeled code corpus, and then to benefit many code-related downstream tasks with fine-tuning. Apart from the denoising sequence-to-sequence objectives widely adopted for pre-training on natural language, identifier tagging and prediction pre-training objective is adopted to enable the model to better leverage the crucial token type information from PL, which specifically are the identifiers assigned by developers.

Systems and methods for code understanding and generation

Embodiments described herein provide training a prompt generator for text classification. A first training dataset associated with a first plurality of class labels is received for a first training process. For a first instance of the first training dataset, a set of labels of interest is generated by sampling from a set of possible class labels including the first plurality of class labels. The prompt generator generates a first prompt based on the set of labels of interest. A pretrained language model generates a task output in response to an input of the first instance prepended with the first prompt. A loss objective is generated based on the task output and the set of labels of interest. Parameters of the prompt generator are updated based on the computed loss function via backpropagation while the pretrained language model is frozen.

Systems and methods for text classification using label modular prompts

Embodiments provide a software framework for evaluating and troubleshooting real-world task-oriented bot systems. Specifically, the evaluation framework includes a generator that infers dialog acts and entities from bot definitions and generates test cases for the system via model-based paraphrasing. The framework may also include a simulator for task-oriented dialog user simulation that supports both regression testing and end-to-end evaluation. The framework may also include a remediator to analyze and visualize the simulation results, remedy some of the identified issues, and provide actionable suggestions for improving the task-oriented dialog system.

Systems and methods for an end-to-end evaluation and testing framework for task-oriented dialog systems

Embodiments described herein provide adversarial attacks targeting the cross-lingual generalization ability of massive multilingual representations, demonstrating their effectiveness on multilingual models for natural language inference and question answering. An efficient adversarial training scheme can thus be implemented with the adversarial attacks, which takes the same number of steps as standard supervised training and show that it encourages language-invariance in representations, thereby improving both clean and robust accuracy.

Systems and methods for code-mixing adversarial training

Embodiments described herein provide systems and methods for an Explicit Memory Tracker (EMT) that tracks each rule sentence to perform decision making and to generate follow-up clarifying questions. Specifically, the EMT first segments the regulation text into several rule sentences and allocates the segmented rule sentences into memory modules, and then feeds information regarding the user scenario and dialogue history into the EMT sequentially to update each memory module separately. At each dialogue turn, the EMT makes a decision among based on current memory status of the memory modules whether further clarification is needed to come up with an answer to a user question. The EMT determines that further clarification is needed by identifying an underspecified rule sentence span by modulating token-level span distributions with sentence-level selection scores. The EMT extracts the underspecified rule sentence span and rephrases the underspecified rule sentence span to generate a follow-up question.

Systems and methods for explicit memory tracker with coarse-to-fine reasoning in conversational machine reading

Embodiments described herein provide an attention-based tree encoding mechanism. Specifically, the attention layer receives as input the pre-parsed constituency tree of a sentence and the lower-layer representations of all nodes. The attention layer then performs upward accumulation to encode the tree structure from leaves to the root in a bottom-up fashion. Afterwards, weighted aggregation is used to compute the final representations of non-terminal nodes.

Systems and methods for a transformer network with tree-based attention for natural language processing

Embodiments described herein provide a dynamic topic tracking mechanism that tracks how the conversation topics change from one utterance to another and use the tracking information to rank candidate responses. A pre-trained language model may be used for response selection in the multi-party conversations, which consists of two steps: (1) a topic-based pre-training to embed topic information into the language model with self-supervised learning, and (2) a multi-task learning on the pretrained model by jointly training response selection and dynamic topic prediction and disentanglement tasks.

Systems and methods for response selection in multi-party conversations with dynamic topic tracking

A visual dialogue model receives image input and text input that includes a dialogue history between the model and a current utterance by a human user. The model generates a unified contextualized representation using a transformer encoder network, in which the unified contextualized representation includes a token level encoding of the image input and text input. The model generates an encoded visual dialogue input from the unified contextualized representation using visual dialogue encoding layers. The encoded visual dialogue input includes a position level encoding and a segment type encoding. The model generates an answer prediction from the encoded visual dialogue input using a first self-attention mask associated with discriminative settings or a second self-attention mask associated with generative settings. Dense annotation fine tuning may be performed to increase accuracy of the answer prediction. The model provides the answer prediction as a response to the current utterance of the human user.

Unified vision and dialogue transformer with BERT

Embodiments described herein provide systems and methods for generating an adversarial sample with inflectional perturbations for training a natural language processing (NLP) system. A natural language sentence is received at an inflection perturbation module. Tokens are generated from the natural language sentence. For each token that has a part of speech that is a verb, adjective, or an adverb, an inflected form is determined. An adversarial sample of the natural language sentence is generated by detokenizing inflected forms of the tokens. The NLP system is trained using the adversarial sample.

Growing community of inventors

Singapore, Singapore

Shafiq Rayhan Joty