International Business Machines Corporation

Twitter, Inc.

Michael Busch

Rajesh M Desai

Fumihiko Terui

Sreeram Viswanath Balakrishnan

Andreas Neumann

Francesco G Callari

Justo Luis Perez

Tom William Jacopi

Magesh Jayapandian

Krishna Gade

Osamu Furusawa

Brian Larson

Frederick Stephen Grennan

David O Been

Ugo Di Girolamo

Robert A Foyle

Patrick J Lok

Samuel Luckenbill

Michael Kyle McCandless

Paul Burstein

Yan Zhao

Christopher John Hinrichs

Yi Zhuang

Brian A Larson

A method and indexing system indexes the content of a body of documents into a content index, and the metadata of the documents into a metadata index which is a parallel index to the content index. The metadata is copied into a data store that is easily accessible by the indexing system and is stored in native form. The indexing system can dynamically re-index the metadata from the native metadata in the data store to produce a new metadata index which is used to replace the original metadata index. Search queries received by a search engine associated with the indexing system are applied to both the content and metadata index and the results are merged for return.

Indexing multiple types of data to facilitate rapid re-indexing of one or more types of data

A system for real-time search, including: a set of partitions, each including a set of segments, each segment corresponding to a time slice of messages posted to the messaging platform, and a real-time search engine configured to receive a search term in parallel with other partitions in set the set of partitions, and search at least one of the set of segments in reverse chronological order of the corresponding time slice to identify document identifiers of messages containing the search term; and a search fanout module configured to: receive a search query including the search term; send the search term to each of the set of partitions for parallel searching; and return, in response to the search query, at least one of the identified document identifiers of messages containing the search term.

Search infrastructure

A system for real-time search, including: a set of partitions, each including a set of segments, each segment corresponding to a time slice of messages posted to the messaging platform, and a real-time search engine configured to receive a search term in parallel with other partitions in the set of partitions, and search at least one of the set of segments in reverse chronological order of the corresponding time slice to identify document identifiers of messages containing the search term; and a search fanout module configured to: receive a search query including the search term; send the search term to each of the set of partitions for parallel searching; and return, in response to the search query, at least one of the identified document identifiers of messages containing the search term.

Methods, systems, and apparatus, including computer programs encoded on computer storage medium, for indexing data in information retrieval systems. In one aspect, a system includes computer readable storage media, where the non-transitory computer readable storage media include postings lists for a term, where each posting list includes postings blocks, position blocks, and metadata blocks. Each postings block includes document identifiers and respective indications of frequency of the term in each document. Each position block includes position indications for the term in the documents of a corresponding postings block. Each metadata block includes an indication of the size of the document identifiers in the associated postings block, an indication of the size of the indications of frequency of terms in the associated postings block, an indication of a number of postings in the associated postings block, and an indication of position information for postings in the associated postings block.

Indexing data in information retrieval systems

Embodiments of the present invention provide a method, system and computer program product for maintaining distributed state consistency in a distributed computing application. In an embodiment of the invention, a method for maintaining distributed state consistency in a distributed computing application can include registering a set of components of a distributed computing application, starting a transaction resulting in changes of state in different ones of the components in the registered set and determining in response to a conclusion of the transaction whether or not an inconsistency of state has arisen amongst the different components in the registered set in consequence of the changes of state in the different ones of the components in the registered set. If an inconsistency has arisen, each of the components in the registered set can be directed to rollback to a previously stored state. Otherwise a committal of state can be directed in each of the components in the registered set.

Managing data consistency between loosely coupled components in a distributed computing system

Provided are techniques for processing a query. A query including constraints for at least two vertically partitioned, inverted indexes is received. The constraints in the query are separated based on the vertically partitioned, inverted indexes. A document identifier iterator is obtained for each of the constraints, wherein each document identifier iterator is associated with a posting list, and wherein each posting list is ordered by document identifier order. A run-time join of the posting lists is performed to obtain a final result set.

Real-time search of vertically partitioned, inverted indexes

Documents and search terms may be received for which an index that supports both incremental document indexing and incremental term indexing is to be generated. From this information, an index may be generated, partitioned in a first dimension by documents to create master segments and in a second dimension by search terms to create slave segments. A request to update the index to include a new document or a new search term may be received. The new document or new search term may be added to the index without modifying the entire index. Further, document identifiers may be synchronized across all segments. Synchronization refers to maintaining consistency of document identifiers across segments, despite renumbering of document identifiers during certain operations such as merging segments.

Parallel segmented index supporting incremental document and term indexing

Techniques are disclosed for searching a plurality of documents using a dynamically defined ignore string. The ignore string may be specified by a user. An overlay index may be generated over the plurality of documents. The overlay index may include a posting list for each term in the ignore string. Each posting list may specify the documents of the plurality of documents in which the respective term occurs outside of the ignore string. The overlay index may also include a posting list that specifies all occurrences of the ignore string in the plurality of documents. Once generated, a user may search the plurality of documents while occurrences of the ignore string in the plurality of documents are ignored in text-based searches.

Searching documents using a dynamically defined ignore string

A method and apparatus are provided for enhancing search results generated by a search engine. A graphical user interface may display search results, which may include at least one multi-part record comprising a plurality of independent documents. Upon receiving a selection of the multi-part record from the search results a detailed view of the record may be displayed. The detailed view of the record may include a listing of the multiple documents that form the record along with an indication of whether keywords from the search criteria are present in the documents. By providing an indication of exactly which documents of a multi-part record include keywords associated with a search, embodiments of the invention allow users to easily identify desired materials from a list of search results comprising the multipart record.

Growing community of inventors

San Jose, CA, United States of America

Michael Busch