Microsoft Technology Licensing, LLC

Yahoo! Inc.

Chad P Walters

John Grant Bennett

Trishul A Chilimbi

Michael Joseph Hopcroft

Jan O Pedersen

Giovanni Lorenzo Thione

Karthik Kalyanaraman

Knut Magne Risvik

Vishesh Parikh

Barney Pell

Lukas A Biewald

Hui Shen

Jianchang Mao

Lintao Zhang

Martin Henk Van Den Berg

Richard Stamford Crouch

Ying Yan

Xiao Wu

Xiaosong Yang

Zhenyu Zhao

Mao Yang

Choong Soon Chang

Nitay Joffe

Andrew Alan James

John JianShi Wang

Daniel Gureasko Bobrow

Zhichen Xu

Daniel Eric Rose

Robert D Cheslow

Brendan O'Connor

Nawaaz Ahmed

Rion Langley Snow

Timothy McDonnell Converse

Scott A Waterman

Robert L Travis, Jr

Albert Meltzer

Methods and computer storage media are provided for generating entries for documents in a forward index. A document and its document identification are received, in addition to static features that are query-independent. The document is parsed into tokens to form a token stream corresponding to the document. Relevant data used to calculate rankings of document is identified and a position of the data is determined. The entry is then generated from the document identification, the token stream of the document, the static features, and the positional information of the relevant data. The entry is stored in the forward index.

Efficient forward ranking in a search engine

Subject matter described herein includes a multi-layer search-engine index. Accordingly, the search-engine index is divided into multiple indexes, each of which includes a respective set of information used to serve (i.e., respond to) a query. One index includes a term index, which organizes a set of terms that are found among a collection of documents. Another index includes a document index, which organizes a set of documents that are searchable. A computing device is used to serve the search-engine index (i.e., to analyze the index when identifying documents relevant to a search query). For example, a solid-state device might be used to serve the multi-layer search-engine index.

Multi-layer search-engine index

Methods and systems are provided for using a hybrid-distribution system to identify relevant documents based on a search query. A group of documents is assigned to a particular segment. The group of documents is indexed both by atom and by document to form a reverse index and a forward index. Both indexes are divided amongst each node in that segment so that each node is responsible for storing and accessing a different portion of both the reverse and forward indexes. The reverse index portion is accessed on each of a first set of nodes to identify a first set of documents that is relevant to a particular search query. Document identifications associated with the first set of documents are used to identify a second set of nodes that access their forward index portions to limit the number of relevant documents to a second set of documents.

Hybrid-distribution model for search engine indexes

Tools and techniques are described herein for checkpointing iterators during search. These tools may provide methods that include instantiating iterators in response to a search request. The iterators include fixed state information that remains constant over a life of the iterator, and further include dynamic state information that is updated over the life of the iterator. The iterators traverse through postings lists in connection with performing the search request. As the iterators traverse the posting lists, the iterators may update their dynamic state information. The iterators may then evaluate whether to create checkpoints, with the checkpoints including representations of the dynamic state information.

Checkpointing iterators during search

Computer-readable media and a computer system for implementing a natural language search using fact-based structures and for generating such fact-based structures are provided. A fact-based structure is generated using a semantic structure, which represents information, such as text, from a document, such as a web page. Typically, a natural language parser is used to create a semantic structure of the information, and the parser identifies terms, as well as the relationship between the terms. A fact-based structure of a semantic structure allows for a linear structure of these terms and their relationships to be created, while also maintaining identifiers of the terms to convey the dependency of one fact-based structure on another fact-based structure. Additionally, synonyms and hypernyms are identified while generating the fact-based structure to improve the accuracy of the overall search.

Fact-based indexing for natural language search

Search results are identified and returned in response to search queries by evaluating and pruning candidate documents in multiple stages. The process employs a search index that indexes atoms found in documents and pre-computed scores for document/atom pairs. When a search query is received, atoms are identified from the search query and a reformulated query is generated based on the identified atoms. The reformulated query is used to identify matching documents, and a preliminary score is generated for matching documents using a simplified scoring function and pre-computed scores in the search index. Documents are pruned based on preliminary scores, and the remaining documents are evaluated using a final ranking algorithm that provides a final set of ranked documents, which is used to generate search results to return in response to the search query.

Matching funnel for large document index

Word sense probabilities are compressed for storage in a semantic index. Each word sense for a word is mapped to one of a number of 'buckets' by assigning a bucket score to the word sense. A scoring function is utilized to assign the bucket scores that maximizes the entropy of the assigned bucket scores. Once the bucket scores have been assigned to the word senses, the bucket scores are stored in the semantic index. The bucket scores stored in the semantic index may be utilized to prune one or more of the word senses prior to construction of the semantic index. The bucket scores may also be utilized to prune and rank the word senses at the time a query is performed using the semantic index.

Efficiently representing word sense probabilities

A role tree having nodes corresponding to semantic roles in a hierarchy is defined. A posting list is generated for each association of a term and a semantic role in the hierarchy. The posting lists are stored contiguously on a physical storage medium such that a subtree of the hierarchy of semantic roles can be loaded from the storage medium as a single contiguous block. The posting lists for a subtree of the hierarchy are retrieved by obtaining data identifying the beginning location on the physical storage medium of the posting lists for the term at the top of a desired subtree of the hierarchy and data identifying the length of the posting lists of the desired subtree of the hierarchy. A single contiguous block that includes the posting lists for the desired subtree of the hierarchy is then retrieved from the beginning location through the specified length.

Efficient storage and retrieval of posting lists

Methods, systems and computer readable media for finding documents in a data store that match a natural language query submitted by a user are provided. The documents and queries are matched by determining that words within the query have the same relationship to each other as the same words in the document. Documents are semantically analyzed and words in the document are indexed along with the role the word plays in a sentence. The initial semantic role may be generalized using a role hierarchy and stored in the index along with the original role. A similar analysis may be used with the search query to find words used in the same role in both the query and the document.

Indexing role hierarchies for words in a search index

Tools and techniques are described that relate to iterators for applying term occurrence-level constraints in natural language searching. These tools may receive a natural language input query, and define term occurrence-level constraints applicable to the input query. The methods may also identify facts requested in the input query, and may instantiate an iterator to traverse a fact index to identify candidate facts responsive to the input query. This iterator may traverse through at least a portion of the fact index. The methods may receive candidate facts from this iterator, with these candidate facts including terms, referred to as term-level occurrences. The methods may apply the term occurrence-level constraints to the term-level occurrences. The methods may select the candidate fact for inclusion in search results for the input query, based at least in part on applying the term occurrence-level constraint.

Iterators for applying term occurrence-level constraints in natural language searching

A system and method for dynamically composing a class is provided. A query is parsed and a compiler object tree is generated that corresponds to the operators in the query. A set of feature requests and responses are initiated and a set of feature objects are generated that are executed at runtime. The language of the query may be extended by registering a new operator, and optionally a new feature for a query language, with a search engine without modifying the source code of the search engine. The new operator may be specified in a plug-in and provided to the search engine via the plug-in. An application converts an end user query into an application query that conforms to the query language supported by the search engine and includes the new operator. The application query may also include references to traditional programming constructs and search engine primitives.

Query serving infrastructure providing flexible and expandable support and compiling instructions

Growing community of inventors

Bellevue, WA, United States of America

Chad P Walters