Microsoft Technology Licensing, LLC

Other

Girish Kumar

Gaurav Sareen

David Benjamin Levitan

Fang Liu

Junhua Wang

Julia S McANALLEN

William Ramsey

Michael Isard

Sanaz Ahari

Ahmad A Abdulkader

Robert Alexander Sim

Nicholas Eric Craswell

Namita Gupta

Farid Hosseini

Qinghua Zou

Pratibha Permandla

Ankur Gupta

Yinzhe Yu

Janine Crumb

Bhuvan Middha

Christopher Lawrence Laterza

Rohit Wad

Silviu Petru Cucerzan

Ryan Frederick Stewart

Vijay Nair

Huseyin Atahan Inan

Xiang Yue

Jiantao Pan

Seyedeh Hoda Shajari

Jason Douglas

Nazan Khan

Charles Lester Alexander Clark

Giridhar Kumaran

Alfian Tan

Improved solutions for dataset clustering and evaluation are disclosed. Examples cluster a set of documents into set of clusters using a language model, in an iterative process. In second and later clustering tasks, the current cluster titles and descriptions are provided in the language model prompt, to avoid near-duplications. Upon determining that the set of clusters is sufficiently complete and representative of the set of documents, the tasking switches to classification of the set of documents into the set of clusters using a language model. Classification continues until a sufficient percentage of the set of documents is classified. Some examples use batching, to avoid overloading the language model(s). In some examples, different language models are used for clustering and classification. Some examples use intruder detection to determine the quality of the clustering. This process provides superior performance on classifying documents having little configuration control, such as website feedback from consumers.

Dataset clustering and evaluation

Systems and methods are directed to building annotated models based on eyes-off data. Specifically, a synthetic data generation model is trained and used to further train a target model. The synthetic data generation model is trained within an eyes-off environment using an anonymity technique on confidential data. The synthetic data generation model is then used to create synthetic data that closely represents the confidential data but without any specific details that can be linked back to the confidential data. The synthetic data is then annotated and used to train the target model within an eyes-on environment. Subsequently, the target model is deployed back within the eyes-off environment to classify the confidential data.

Building annotated models based on eyes-off data

A method and system for generating synthetic privacy preserving training data for training a language classifier machine-learning (ML) model includes receiving a request to generate the synthetic privacy-preserving training data for the language classifier ML model, retrieving labeled training data associated with training the language classifier ML model, providing the labeled training data, one or more privacy parameters, and a domain type associated with the labeled training data to a synthetic data generation ML model, the synthetic data generation ML model being configured to generate synthetic training data in a privacy-persevering manner, receiving synthetic privacy-preserving training data as an output from the synthetic data generation ML model, and providing the synthetic privacy preserving training data to the language classifier ML model for training the language classifier ML model in classifying text.

Generating private synthetic training data for training machine-learning models

Computer-readable media, computer systems, and computing methods are provided for classifying search results as either of good quality or of poor quality. Initially, a portion of the search results, such as the highest ranked documents, are selected for evaluation. A level of quality for each of the selected search results is determined using a classification process that includes the following steps: targeting features demonstrated by the selected search results to be evaluated; evaluating the selected features to generate a level-of-quality score for each of the selected search results; comparing the score against a predefined threshold value; and, based on the comparison, assigning each of the selected search results an absolute measurement. The absolute measurement indicates poor quality when the score is less than the threshold value. Upon recognizing that the selected search results are of poor quality, automatically executing a corrective action that reformulates the issued search query.

Classifying results of search queries

Subject matter described herein is related to determining a document score, which suggests a relevance of a document (e.g., webpage) to a search query. For example, a search query is received that is comprised of one or more terms, which represent a subject. An equivalent subject is identified that is semantically similar to the subject. The document score is determined by accounting for both a subject frequency and an equivalent-subject frequency.

Information retrieval using subject-aware document ranker

Techniques are disclosed for providing a domain-aware snippet for a search result. A uniform resource locator (URL) is identified for a search result obtained in response to a search query, and it is determined that the URL corresponds to a single domain that has a plurality of web pages that are generated using a template that is common to each of the web pages in the domain. The template comprises a hypertext markup language (HTML) layout pattern that includes multiple sections shared by the web pages. A ranking value is assigned to the multiple sections and is used to identify a first section of the template that is relevant to the search query. A snippet is provided to a user for the search result; the snippet includes at least a portion of text from the first section.

Domain-aware snippets for search results

A search engine can provide referencing information as context for a particular search result when an excerpt from the search result, comprising at least some similar elements to the user's query, is not generated. Referencing information can include one or more anchor texts having similarity to at least some elements of the user's query, the anchor texts being used by referencing pages to link to the page returned as a search result. User selection of the anchor text can enable the user to visit a referencing page using that anchor text to link to the page returned as a search result, and having a high static rank.

Using anchor text to provide context

Techniques are disclosed for providing a domain-aware snippet for a search result. With such techniques, a domain classification component is provided for identifying a template used to generate a plurality of web pages of a domain, associating the template and content of the web pages related to the template with a Uniform Resource Locator pattern of the plurality of web pages, and storing the associated template, the related content, and the Uniform Resource Locator pattern in a database. A snippet extraction component is also provided for extracting text from a section of a web page of the plurality of web pages for a snippet of a search result corresponding to a search query, wherein the extracted text is based on a ranking value of the section and the relevance of the extracted text to the search query.

Systems, methods, computer-readable media, and graphical user interfaces for presenting search results having collapsed domains are provided. A search result obtaining module obtains search results based upon a received query. Upon obtaining the search results, search results having the same domain are associated with one another. Thereafter, search result clusters of associated search results are formed. In some embodiments, the search result clusters may be formatted to include desired search result cluster attributes. The search result clusters are presented such that two or more associated search results form a single cluster of search results rather than being presented individually. In some embodiments, an option to view more search results with the same domain may be provided.

Domain collapsing of search results

Computer-readable media, systems, and methods for augmenting URL queries are described. In embodiments, a URL query is received from a user and it is determined whether the URL query is a simple URL query. Further, if the URL query is a simple URL query, an augmented query is created by word-breaking at least a portion of the URL query and the augmented query is associated with one or more ranking preferences. In various other embodiments, a URL query is received from a user and it is determined whether the URL query is a complex URL query. Further, if the URL query is a complex URL query, an augmented query is created that is identical to the URL query and the augmented query is associated with one or more ranking preferences.

Augmenting URL queries

The embodiments contemplate a system and method for obtaining related results for a portion of a query and for generating an updated set of queries for a cache of a server. Other queries beginning with the portion of the query may be identified and obtained from a data structure that includes a server cache and a set of common queries. Once the other queries are identified, results for the other queries are obtained from the server cache or from a back-end database. A set of common queries, which may include deleted and additional queries, may be utilized to generate the updated set of queries for the server. Both missing queries and deleted queries that may belong to the server based on an assignment function are inserted into a queue, which is later delivered to the cache of the server. The transfer may occur during a low-activity or idle state.

Serving cached query results based on a query portion

Computerized methods and systems for generating a suggested query list with suggested search terms displayed as highlighted text utilizing a user-defined query are provided. Query search terms are received by a user-interface display. Upon inputting query search terms, the user-interface automatically generates a suggested query list. The suggested query list is associated with the query search term and the suggested query list is comprised of at least one suggested search term. A query suggestion architecture determines if the query search term and the suggested search term are a match, and if so, highlights the suggested search term that is not a match. The user interface displays the highlighted terms to assist in refining a search. The present invention further provides a stemming algorithm that extracts the root form of the query search term.

Support for reverse and stemmed hit-highlighting

The claimed subject matter relates to a computer-implemented architecture that can, at a high level, store query results in a location-independent manner in order to facilitate caching of local results. To store query results in a location-independent manner such that cached results to location-based queries can be useful, the architecture can further include a mechanism for encoding a surface or area (e.g., the earth) based upon document density rather than geography. The encoding mechanism can also organize an inverted index so that no join operation is required to return valid results to a location-based query. The architecture can also include a mechanism for determining when previously cached results are adequate to satisfy a query.

Growing community of inventors

Redmond, WA, United States of America

Girish Kumar