Microsoft Technology Licensing, LLC

Silicon Graphics, Incorporated

Michael Joseph Hopcroft

Trishul A Chilimbi

Karthik Kalyanaraman

Knut Magne Risvik

Robert Lovejoy Goodwin

John Grant Bennett

Paul R Erickson

Vishesh Parikh

Jonathan David Perlow

Jan O Pedersen

Chad P Walters

Nancy Ellen Jacobs

Gavin Morris Kelly

Eric S Rockey

Shannon P Talbott

Daniel J Westreich

Utkarsh Jain

Fan Wang

Denis V Deyneko

Andrija Antonijevic

Satoshi Nakajima

Oliver Griffith Fisher

Richard J Wolf

Suryanarayanan V Raman

Michael C Murray

Scott L Ruthfield

Brian Keith Cabral

Hui Shen

Christopher Walter Anderson

Henry Setiawan

Antonia Spyridi

Methods and computer storage media are provided for generating entries for documents in a forward index. A document and its document identification are received, in addition to static features that are query-independent. The document is parsed into tokens to form a token stream corresponding to the document. Relevant data used to calculate rankings of document is identified and a position of the data is determined. The entry is then generated from the document identification, the token stream of the document, the static features, and the positional information of the relevant data. The entry is stored in the forward index.

Efficient forward ranking in a search engine

The technology described herein provides for identifying matching documents for a search query using a bit vector search index. When a search query is received, a term is identified from the search index, and a number of bit vectors corresponding to the term are identified. Each bit vector comprises an array of bits in which at least one bit in each bit vector indicates that a corresponding document includes the term. Each bit vector also includes other bits indicating other documents include other terms. A determination is made that an unacceptable number of possible matching documents is likely to be returned. In response to the determination, a strengthening row bit vector is selected to reduce the number of possible matching documents. The identified bit vectors and the selected strengthening row are intersected to identify matching documents that contain the term.

Reducing matching documents for a search query

The technology described herein provides for a match fix-up stage that removes matching documents identified for a search query that don't actually contain terms from the search query. A representation of each document (e.g., a forward index storing a list of terms for each document) is used to identify valid matching documents (i.e., documents containing terms from the search query) and invalid matching documents (i.e., documents that don't contain terms from the search query). Any invalid matching documents are removed from further processing and ranking for the search query.

Match fix-up to remove matching documents

The technology described herein provides for preliminary ranking of matching documents for a search query. A preliminary ranker uses score tables for scoring each matching document based on its relevant to a search query. The score table for a document stores pre-computed data used to derive a frequency of terms and other information in the document. The preliminary ranker uses the score table for each matching document and the terms from the search query to determine a score for each matching document. The lowest scoring documents are removed from further consideration by a final ranker.

Preliminary ranker for scoring matching documents

The technology described herein provides for indexing information in a bit vector search index. The bit vector search index comprises a data structure for indexing data about terms from a corpus of documents. The data structure includes a number of bit vectors. Each bit vector comprises an array of bits and corresponds to a different set of terms. Bits in the bit vector are used to represent whether at least one document corresponding to the bit includes at least one term from the set of terms corresponding to the bit vector. The bit vector search index is stored by first indexing information about documents using bit vectors on a first accumulation buffer storage device. When a threshold is satisfied, the information is transferred to bit vectors on a subsequent storage device.

Updating a bit vector search index

The technology described herein provides a bit vector search index for a search system that uses shards. The bit vector search index comprises a data structure for indexing data about terms from a corpus of documents. The data structure includes a number of bit vectors. Each bit vector comprises an array of bits and corresponds to a different set of terms. Bits in the bit vector are used to represent whether at least one document corresponding to the bit includes at least one term from the set of terms corresponding to the bit vector. The search index is provided in a number of shards. Each shard corresponds to a subset of documents having documents lengths within particular a range of document lengths.

Bit vector search index using shards

The technology described herein provides for identifying matching documents for a search query using a bit vector search index. When a search query is received, a term is identified from the search index, and a number of bit vectors corresponding to the term are identified. Each bit vector comprises an array of bits in which at least one bit in each bit vector indicates that a corresponding document includes the term. Each bit vector also includes other bits indicating other documents include other terms. The identified bit vectors are intersected to identify matching documents that contain the term.

Matching documents using a bit vector search index

The technology described herein provides for preliminary ranking of matching documents for a search query. A preliminary ranker uses score tables for scoring each matching document based on its relevant to a search query. The score table for a document stores pre-computed data used to derive a frequency of terms and other information in the document. The preliminary ranker uses the score table for each matching document and the terms form the search query to determine a score for each matching document. The lowest scoring documents are removed from further consideration by a final ranker.

The technology described herein provides for storing and retrieving data in a bit vector search index. The bit vector search index stores data about terms from documents using bit vectors. Each bit vector comprises an array of bits and corresponds to a different set of terms. Each bit in the bit vector is used to represent whether a document includes at least one term from the set of terms. A band table is used to store bit vector configurations for bands of terms having similar term characteristics. Each term is indexed in the bit vector search index according to a bit vector configuration for a band to which it belongs. When identifying bit vector storage locations for terms, explicit mappings are used for some terms and ad hoc approaches used for other terms. Explicit mappings provide specific locations for terms, while ad hoc approaches use mapping algorithms assigned to bands.

Storage and retrieval of data from a bit vector search index

A search index includes tiered posting lists. Each posting list in the search index corresponds with a different atom and includes a list of documents containing the particular document. Additionally, a rank is stored with each document listed in a posting list for a given atom representing the relevance of the atom to the context of each document. At least some of the posting lists in the search index are tiered. A tiered posting list is divided into a number of tiers with the tiers being ordered by document while each tier is internally ordered by document. Employing tiered posting lists within the search index allows a search engine to evaluate search queries in a manner that allows for a number of efficiencies and precise stopping.

Tiering of posting lists in search engine index

Methods and systems present commands to a user within a software application program by determining the user's context within the application program and automatically presenting in a user interface commands that pertain to the user's current context. When the user's context changes, the context-sensitive commands are automatically removed from the user interface. In one implementation context blocks and context panes are employed to present the commands.

Task-sensitive methods and systems for displaying command sets

Methods and systems are provided for using a hybrid-distribution system to identify relevant documents based on a search query. A group of documents is assigned to a particular segment. The group of documents is indexed both by atom and by document to form a reverse index and a forward index. Both indexes are divided amongst each node in that segment so that each node is responsible for storing and accessing a different portion of both the reverse and forward indexes. The reverse index portion is accessed on each of a first set of nodes to identify a first set of documents that is relevant to a particular search query. Document identifications associated with the first set of documents are used to identify a second set of nodes that access their forward index portions to limit the number of relevant documents to a second set of documents.

Hybrid-distribution model for search engine indexes

A preliminary segment root and a final segment root are selected for each segment. Each time a search query is received, a set of nodes in each segment that will be used to resolve the search query is identified. A preliminary segment root is selected from the set of nodes. Based on statistical data from each node in the set of nodes indicating each node's capability to act as a final segment root that assembles query-execution data, the preliminary segment root algorithmically selects the final segment root. The other nodes in the set of nodes are notified regarding the identity of the final segment root.

Growing community of inventors

Kirkland, WA, United States of America

Michael Joseph Hopcroft