Microsoft Technology Licensing, LLC

Weizhu Chen

Zheng Chen

Pengcheng He

Gang Wang

Jianfeng Gao

Zhikai Fan

Jian Wang

Benyu Zhang

Xiaodong Liu

Michael Nelson

Woo Ram Lee

Matthew Brigham Hall

Ganapathy V Raman

Jianwen Zhang

Yu Zhao

Santosh Vasant Bhosale

Manoj Ajith Prasad

Sparky Toews

Aritra DattaGupta

Aarohi Arora

Yuting Sun

Will Qian

Roger F Johnson

Vladyslav Bazyliak

Somak Mitra

Bryan Wilhelm

Benjamin Reich

Matthew Justice

Xuedong D Huang

Dong Li

Ming Zhou

Jun Yan

Ning Liu

Min Wang

Yi-Min Wang

Yu Wang

Ralf Herbrich

Qian Li

Jingren Zhou

Jian-Tao Sun

Lei Ji

Long Jiang

Hoifung Poon

Zhimin Zhang

Dou Shen

Chenguang Zhu

Wei Lin

Thomas Minka

Eslam K Abdelreheem

Adish Singla

Yelong Shen

Jiayuan Huang

Thore Kurt Hartwig Graepel

Mingyu Wang

Junyan Chen

Po-Sen Huang

Thomas Ivan Borchert

Joaquin Quinonero Candela

Hao Cheng

Katherine Winant Osborne

Vikas Bahirwani

Jade Huang

Robert S Harrell

Liaoyuan Ning

Jiaye Xie

Taha Bekir Eren

Oleg Isakov

Jeffrey Scott Dunn

Li Ju

Generally discussed herein are devices, systems, and methods for generating an embedding that is both local string dependent and global string dependent. The generated embedding can improve machine learning (ML) model performance. A method can include converting a string of words to a series of tokens, generating a local string-dependent embedding of each token of the series of tokens, generating a global string-dependent embedding of each token of the series of tokens, combining the local string dependent embedding the global string dependent embedding to generate an n-gram induced embedding of each token of the series of tokens, obtaining a masked language model (MLM) previously trained to generate a masked word prediction, and executing the MLM based on the n-based induced embedding of each token to generate the masked word prediction.

ML using n-gram induced input representation

This document relates to training of machine learning models. One example method involves providing a machine learning model having one or more mapping layers. The one or more mapping layers can include at least a first mapping layer configured to map components of pretraining examples into first representations in a space. The example method also includes performing a pretraining stage on the one or more mapping layers using the pretraining examples. The pretraining stage can include adding noise to the first representations of the components of the pretraining examples to obtain noise-adjusted first representations. The pretraining stage can also include performing a self-supervised learning process to pretrain the one or more mapping layers using at least the first representations of the training data items and the noise-adjusted first representations of the training data items.

Adversarial pretraining of machine learning models

Methods, systems, apparatuses, and computer program products are provided for extracting an entity value from a sentence. An embedding set that may include one or more sentence embeddings is generated for at least part of a first sentence that is tagged to associate a first named entity in the sentence with an entity type. A plurality of candidate embeddings is also generated for at least part of a second sentence. The one or more sentence embeddings in the embedding set may be compared with each of the plurality of candidate embeddings, and a match score may be assigned to each comparison to generate a match score set. A particular match score of the match score set may be identified that exceeds a similarity threshold, and an entity value of the entity type may be extracted from the second sentence associated with the identified match score.

Example based entity extraction, slot filling and value recommendation

Techniques for iterative query-based analysis of text are described. According to various implementations, a neural network architecture is implemented receives a query for information about text content, and iteratively analyzes the content using the query. During the analysis a state of the query evolves until it reaches a termination state, at which point the state of the query is output as an answer to the initial query.

Iterative query-based analysis of text

Systems and methods are provided for facilitating the building and use of natural language understanding models. The systems and methods identify a plurality of tokens and use them to generate one or more pre-trained natural language models using a transformer. The transformer disentangles the content embedding and positional embedding in the computation of its attention matrix. Systems and methods are also provided to facilitate self-training of the pre-trained natural language model by utilizing multi-step decoding to better reconstruct masked tokens and improve pre-training convergence.

Efficient transformer language models with disentangled attention and multi-step decoding

Conversational virtual assistance for delivering relevant query solutions is provided. A virtual assistant system comprises various components associated with developing a knowledge database that can be searched for finding documents that fulfill the user's intent. The virtual assistant system further comprises components for receiving a query from a user, extracting entities for understanding the user's intent, and for searching a knowledge database for documents responsive to the query. When additional information is needed for determining more relevant results, a conversation strategy is determined, and a question is formulated for generating a conversation with the user for clarifying the user's intent, confirming a solution, or obtaining additional information. The user is enabled to provide a follow-up response that is related to a previously identified entity. The entity is edited in the query, and responses are refined responsive to the edited query.

Conversational virtual assistant

Methods for configurable and proactive application diagnostics and recovery are performed by systems and devices. A diagnostics manager determines diagnostics packages corresponding to problems described in client device diagnostics requests. Session identifiers are generated and returned with diagnostics identifiers to clients which then provide the session identifiers and diagnostics identifiers to a service manager for session initiation. Diagnostics packages are located, retrieved, and provided back to the client by the service manager that invokes a client-side engine to execute diagnostics packages. Results are provided to the diagnostics system which determines additional packages to be executed by the engine during the same diagnostics session. Further, device-specific tokens are acquired by client devices which execute local diagnostic packages and acquire remote diagnostic packages for execution in the same session. Additionally, diagnostics packages provide their own user interfaces, and local packages are automatically executed upon invocation of help applications.

System and method for configurable and proactive application diagnostics and recovery

A system for training and deploying an artificial conversational entity using an artificial intelligence (AI) based communications system is disclosed. The system may comprise a memory storing machine readable instructions. The system may also comprise a processor to execute the machine readable instructions to receive a request via an artificial conversational entity. The processor may also transmit a response to the request based on a dialog tree generated from at least a model-based action generator and a memory-based action generator. The processor may further provide a training option to a user in the event the response is suboptimal. The processor may additionally receive a selection from the user via the training option. The selection may be associated with an optimal response.

Interactive dialog training and communication system using artificial intelligence

Generally discussed herein are devices, systems, and methods for visualization of user intent in accessing a virtual agent. A method can include receiving sentences from the respective interaction sessions, projecting the sentences to a higher-dimensional space to create respective higher-dimensional vectors, projecting the higher-dimensional vectors to a lower-dimensional space to create respective lower-dimensional vectors, the lower-dimensional space including either two dimensions or three dimensions, plotting the lower-dimensional vectors as respective points on a graph, encoding the respective points consistent with the respective intents determined to be associated with the sentences by an intent classifier to create encoded points, and generating a visual representation of the encoded points.

Visualization of user intent in virtual agent interaction

A method for machine learning a data set in a data processing framework is disclosed. A forest is trained with the data set that generates a plurality of trees in parallel. Each tree includes leaf nodes having a constant weight. A discriminative value for each leaf node is learned with a supervised model. The forest is reconstructed with the discriminative values replacing the constant weight for each leaf node.

Distributed stage-wise parallel machine learning

The techniques discussed herein consider a degree of noise associated with user clicks performed during search sessions. The techniques then generate a model that characterizes click noise so that search engines can more accurately infer document relevance.

Click noise characterization model

Methods, systems, and media are provided for a dynamic batch strategy utilized in parallelization of online learning algorithms. The dynamic batch strategy provides a merge function on the basis of a threshold level difference between the original model state and an updated model state, rather than according to a constant or pre-determined batch size. The merging includes reading a batch of incoming streaming data, retrieving any missing model beliefs from partner processors, and training on the batch of incoming streaming data. The steps of reading, retrieving, and training are repeated until the measured difference in states exceeds a set threshold level. The measured differences which exceed the threshold level are merged for each of the plurality of processors according to attributes. The merged differences which exceed the threshold level are combined with the original partial model states to obtain an updated global model state.

Parallelization of online learning algorithms

A smart user-centric information aggregation system allows a user to define a region of content displayed in a display of a device and performs information aggregation on behalf of the user. The smart user-centric information aggregation system searches, aggregates and groups information related to content included in the region of content for the user while the user can continue to perform his/her original course of actions without interruption. After finding information related to the desired content, the smart user-centric information aggregation system may notify the user and present the found information to the user upon receiving confirmation from the user. The smart user-centric information aggregation system may continue to find new related information and update the presentation with the newly found information periodically, in some instances without user intervention or input.

Smart user-centric information aggregation

User behavior modeling can include determining actions performed by various users. From the mined user actions, future actions can be predicted. Certain implementations include providing information and/or services based on the predicted future actions. Some implementations, include providing relevant information, services, and/or goods regarding the predicted future action.

Action prediction and identification of user behavior

Growing community of inventors

Beijing, China

Weizhu Chen