Microsoft Technology Licensing, LLC

Utkarsh Jain

Fan Wang

Robert Lovejoy Goodwin

Michael Joseph Hopcroft

Jay Goyal

Neil Sharman

Denis V Deyneko

Vibhaakar Sharma

Vinay Deshpande

Daniel Yuan

Andrija Antonijevic

Gaurav Sareen

Yinzhe Yu

Xingtao Wei

Syed Jawad

Martin Irman

Oren Firestein

The technology described herein provides for identifying matching documents for a search query using a bit vector search index. When a search query is received, a term is identified from the search index, and a number of bit vectors corresponding to the term are identified. Each bit vector comprises an array of bits in which at least one bit in each bit vector indicates that a corresponding document includes the term. Each bit vector also includes other bits indicating other documents include other terms. A determination is made that an unacceptable number of possible matching documents is likely to be returned. In response to the determination, a strengthening row bit vector is selected to reduce the number of possible matching documents. The identified bit vectors and the selected strengthening row are intersected to identify matching documents that contain the term.

Reducing matching documents for a search query

The technology described herein provides for a match fix-up stage that removes matching documents identified for a search query that don't actually contain terms from the search query. A representation of each document (e.g., a forward index storing a list of terms for each document) is used to identify valid matching documents (i.e., documents containing terms from the search query) and invalid matching documents (i.e., documents that don't contain terms from the search query). Any invalid matching documents are removed from further processing and ranking for the search query.

Match fix-up to remove matching documents

The technology described herein provides for indexing information in a bit vector search index. The bit vector search index comprises a data structure for indexing data about terms from a corpus of documents. The data structure includes a number of bit vectors. Each bit vector comprises an array of bits and corresponds to a different set of terms. Bits in the bit vector are used to represent whether at least one document corresponding to the bit includes at least one term from the set of terms corresponding to the bit vector. The bit vector search index is stored by first indexing information about documents using bit vectors on a first accumulation buffer storage device. When a threshold is satisfied, the information is transferred to bit vectors on a subsequent storage device.

Updating a bit vector search index

In various embodiments, systems and methods are provided for generating and using a customized index. In embodiments, an index structure is constructed to efficiently utilize machines containing index portions. In this regard, the index structure for a particular application is customizable such that a number of virtual index units for a particular index type and/or a number of machines associated with the virtual index units for the particular index type can be optimized for machine and/or system performance and efficiency. Utilizing the constructed index structure, documents can be distributed to various index units, virtual index units, and/or machines in real-time or near real-time. Further, the customized index structure can be used to efficiently serve search results in response to search queries.

Generating and using a customized index

Methods and systems for quickly serving documents are provided. Documents may be served to users, for example, in response to search query inputs. Documents may be individually communicated to a document server prior to batching the documents. By individually communicating documents to document servers, the document experiences sub-second latency before it is available to a user. The documents may also be modified individually such that real-time serving is not interrupted.

Receiving individual documents to serve

Methods and systems for quickly serving documents are provided. Documents may be served to users, for example, in response to search query inputs. Documents may be individually communicated to a document server individually prior to batching the documents. In such a real-time serving system, serving components may fail. To ensure real-time serving despite the failure, spares are utilized to replace the failing serving components such that the spare can immediately begin receiving documents. The spare can also be synchronized with other serving components to obtain the memory of the failing serving component prior to the failure.

Recovery of a document serving environment

Methods, systems, and media are provided for an optimized search engine index. The optimized index is formed by merging small lower level indexes of fresh documents together into a hierarchical cluster of multiple higher level indexes. The optimized index of fresh documents is formed via a single threaded process, while a fresh index serving platform concurrently serves fresh queries. The hierarchy of higher level indexes is formed by merging lower and/or higher level indexes with similar expiration times together. Therefore, as some indexes expire, the remaining un-expired indexes can be re-used and merged with new incoming indexes. The single threaded process provides fast serving of fresh documents, while also providing time to integrate the fresh indexes into a long term primary search engine index, prior to expiring.

Growing community of inventors

Bellevue, WA, United States of America

Utkarsh Jain