International Business Machines Corporation

Stratify, Inc.

Hewlett Packard Enterprise Development LP

Sauraj Goswami

You-Chin Gene Fuh

James Zu-Chia Teng

Terence Patrick Purcell

Thomas Abel Beavin

Andrey Balmin

Michael R Shadduck

Balakrishna Raghavendra Iyer

Jeffrey William Josten

Joy Aloysius Thomas

Jeffrey Allen Berger

Vamsi Salaka

Stephen Walter Turnbaugh

Documents likely to be near-duplicates are clustered based on document vectors that represent word-occurrence patterns in a relatively low-dimensional space. Edit distance between documents is defined based on comparing their document vectors. In one process, initial clusters are formed by applying a first edit-distance constraint relative to a root document of each cluster. The initial clusters can be merged subject to a second edit-distance constraint that limits the maximum edit distance between any two documents in the cluster. The second edit-distance constraint can be defined such that whether it is satisfied can be determined by comparing cluster structures rather than individual documents.

Clustering of near-duplicate documents

Multiple nonoverlapping languages within a single document can be identified. In one embodiment, for each of a set of candidate languages, a set of non-overlapping languages is defined. The document is analyzed under the hypothesis that the whole document is in one language and that part of the document is in one language while the rest is in a different, non-overlapping language. Language(s) of the document are identified based on comparing these competing hypotheses across a number of language pairs. In another embodiment, transitions between non-overlapping character sets are used to segment a document, and each segment is scored separately for a subset of candidate languages. Language(s) of the document are identified based on the segment scores.

Language identification for documents containing multiple languages

An 'impostor profile' for a language is used to determine whether documents are in that language or no language. The impostor profile for a given language provides statistical information about the expected results of applying a language model for one or more other ('impostor') languages to a document that is in fact in the given language. After a most likely language for a test document is identified, the impostor profile is used together with the scores for the test document in the various impostor languages to determine whether to identify the test document as being in the most likely language or in no language.

Automated identification of documents as not belonging to any language

A query of at least one mark-up language document has a path expression comprising a conjunction, a first filter and a second filter. The first filter has a first probe. The second filter has a second probe. The first and second filters form a between filter having start and stop values specified by the first and second probes. A plan to process the query is generated based on, at least in part, a range defined by the start and stop values. An index of mark-up language documents is defined by another path expression; the index comprises values of mark-up language documents that satisfy the other path expression; the values are key values of the index. The plan is to perform a single scan of the key values from the start value to the stop value to identify at least one key value that satisfies the between filter.

Between matching

A system for executing a query on data that has been partitioned into a plurality of partitions is provided. The system includes providing partitioned data including one or more columns and the plurality of partitions. The partitioned data includes a limit key value associated with each column for a given partition. The system further includes receiving a query including a predicate on one of the one or more columns of the partitioned data; and utilizing the predicate on the one of the one or more columns in a pruning decision on at least one of the one or more partitions based on the limit key values associated with the plurality of partitions.

Generalized partition pruning in a database system

Various embodiments of a computer-implemented method, computer program product, and data processing system are provided that generate an index plan that produces a superset of data comprising the query result. In some embodiments, a computer-implemented method, computer program product, and data processing system produce a maximal-index-satisfiable query tree.

Index exploitation

Various embodiments of a computer-implemented method, system and computer program product maintain a logical page having a predetermined size. Data is added to an uncompressed area of the logical page. The uncompressed area of the logical page is associated with an uncompressed area of a physical page. The logical page also has a compressed area associated with a compressed area of a physical page. In response to exhausting the uncompressed area, data in the uncompressed area is included in the compressed area. The uncompressed area is adjusted.

Compressibility checking avoidance

Various embodiments of a computer-implemented method, system and computer program product are provided. A first plurality of key entries of a first index page are compressed in accordance with an order specified by a first keymap of the first index page. The first keymap also indicates respective positions of the key entries of the first plurality of key entries. A second keymap is generated indicating the order and also indicating respective post-compression positions of the key entries of the first plurality of key entries. The compressed first plurality of key entries is stored on a second index page with the second keymap.

Keymap order compression

A partial index availability system places, in a restricted state, all pages in the index associated with a structure modification, when an error occurs in processing a log of the said structure modification. This maintains traversability of the rest of the index that is not in restricted state. The system locates and marks a left sentinel and a right sentinel associated with a non-leaf page that is in a restricted state preventing an undo of a transaction. The sentinels prevent a transaction from accessing an uncommitted change associated with the non-leaf page. After a recovery procedure is run the entire index is made available. During the period between the placement of the index pages in LPL or rebuild pending to the time of final removal of these pages from their restrictive states as a result of a recovery procedure being run, the users are given access to the non-restricted portion of the index.

System and method for increasing availability of an index

Methods for executing a query on data that has been partitioned into a plurality of partitions are provided. The method includes providing partitioned data including one or more columns and the plurality of partitions. The partitioned data includes a limit key value associated with each column for a given partition. The method further includes receiving a query including a predicate on one of the one or more columns of the partitioned data; and utilizing the predicate on the one of the one or more columns in a pruni.ng decision on at least one of the one or more partitions based on the limit key values associated with the plurality of partitions.

Growing community of inventors

Palo Alto, CA, United States of America

Sauraj Goswami