International Business Machines Corporation

Frederick J Damerau

David E Johnson

Sholom M Weiss

Chidanand V Apte

Martin C Buskirk, Jr

Robert L Mercer

Brian F White

Eric K Mays

Marguerite Raaen

David H Johnson

A procedure automates the process of setting up an instance of a conversational natural language interface for a Web site. By automating the process of setting up a new Web site, the process enables a new interface to be created by anyone. Subsequent manual tuning of the interface is possible and much easier to do than creating an interface from scratch. In order to set up an instance of a natural language conversational interface, it is necessary to define a hierarchy of topics into which individual documents or Web pages can be classified, provide a keyword index for those documents for an associated search engine, and for each node in the hierarchy, specify a mechanism for associating an input natural language (NL) query to the node.

Automated set up of web-based natural language interface

A method and apparatus for providing summaries of documents belonging to a class of documents in a classified document collection. A sample set of documents belonging to one or more classes is processed via a machine learning system in order to induce a set of rules associated with the sample set of documents. The vocabulary in the rules are extracted and compared to words, terms or phrases of an incoming document. Any matches between the extracted rules and the words, terms or phrases of the incoming document are used as a summary for the incoming document. By using the method and apparatus, each document does not have to be processed to find most important words and the like in order to provide a summary for that document and then repeating the same process for additional documents.

Rule induction for summarizing documents in a classified document collection

A method of automatically labeling of unlabeled text data can be practiced independent of human intervention, but that does not preclude manual intervention. The method can be used to extract relevant features of unlabeled text data for a keyword search. The method of automated labeling of unlabeled text data uses a document collection as a reference answer set. Members of the answer set are converted to vectors representing centroids of unknown groups of unlabeled text data. Unlabeled text data are clustered relative to the centroids by a nearest neighbor algorithm and the ID of the relevant answer is assigned to all documents in the cluster. At this point in the process, a supervised machine learning algorithm is trained on labeled data, and a classifier for assigning labels to new text data is output. Alternatively, a feature extraction algorithm may be run on classes generated by the step of clustering, and search features output which index the unlabeled text data.

Automatic labeling of unlabeled text data

A rules based configurable system efficiently and effectively determines for a given electronically represented text document which linguistic analysis and extraction processes and which application specific processes should be invoked to provide more accurate answers to a user's query. In a rules based classifier, where each category or topic is represented by a set of rules, in an application such as routing, the categorization effecting the routing can be effectively combined with processes extracting other information. This may be in the form of a prompt for the user to input additional information.

Categorization based text processing

A machine learning based electronic mail system. A classifier and action selection module analyzes the incoming message and classifies the messages with associated confidence levels, which may include analyzing the electronic message by tokenization of the text, morphological analysis of the text, and other well known processes. The classifier and action selection module then determines the appropriate action or actions to effect on the message.

Machine learning based electronic messaging system

A lightweight document matcher employs minimal processing and storage. The lightweight document matcher matches new documents to those stored in a database. The matcher lists, in order, those stored documents that are most similar to the new document. The new documents are typically problem statements or queries, and the stored documents are potential solutions such as FAQs (Frequently Asked Questions). Given a set of documents, titles, and possibly keywords, an automatic back-end process constructs a global dictionary of unique keywords and local dictionaries of relevant words for each document. The application front-end uses this information to score the relevance of stored documents to new documents. The scoring algorithm uses the count of matched words as a base score, and then assigns bonuses to words that have high predictive value. It optionally assigns an extra bonus for a match of words in special sections, e.g., titles. The method uses minimal data structures and lightweight scoring algorithms to compute efficiently even in restricted environments, such as mobile or small desktop computers.

Light weight document matcher

A text categorization method automatically classifies electronic documents by developing a single pooled dictionary of words for a sample set of documents, and then generating a decision tree model, based on the pooled dictionary, for classifying new documents. Adaptive resampling techniques are applied to improve the accuracy of the decision tree model.

Method for improvement accuracy of decision tree based text categorization

A method and apparatus for determining whether a record, or an edited version thereof, is stored in a computer system. With this invention, whenever a record is stored in the system a hash function is applied to subsets of a key representing the record to be stored to generate multiple hash addresses. A copy of the key, or pointer thereto, is stored at each of the generated hash addresses. Whenever one wishes to determine whether a key is stored in the system, a hash function is applied to subsets of the test record to generate multiple hash addresses. The key for the test record then compared with the key stored in each of the generated hash addresses. If the key for the test record is sufficiently close to anyone of the keys found at the hash addresses, the test record is assumed to be stored in the system.

Storing and retrieving records in a computer system

A method of detecting and correcting an error in a string of information signals. When each information signal represents a word, the method detects and corrects spelling errors. The method detects and corrects an error which is a properly spelled word, but which is the wrong (not intended) word. For example, the method is capable of detecting and correcting a misspelling of 'HORSE' as 'HOUSE'. In the spelling error detection and correction method, a first word in an input string of words is changed to form a second word different from a first word to form a candidate string of words. The spellings of the first word and the second word are in the spelling dictionary. The probability of occurrence of the input string of words is compared to the product of the probability of occurrence of the candidate string of words multiplied by the probability of misrepresenting the candidate string of words as the input string of words. If the former is greater than or equal to the latter, no correction is made. If the former is less than the latter, the candidate string of words is selected as a spelling correction.

Method and apparatus for 'wrong word' spelling error detection and

A guess-ahead feature for an interactive terminal having a keyboard and a display screen where input data is entered via the keyboard and displayed. Means are provided for continually evaluating input data to determine if it is the beginning of a string of data stored in the system memory. If the input data is determined to match the beginning of the string of prestored data, the complete string of stored data is displayed without moving the cursor. A function key is provided so that if the displayed complete string is the string the terminal operator desires to enter, the terminal operator can, by the depressing the function key, advance the cursor to the end of the string. If, however, the displayed string is not exactly as desired, the operator merely continues keying input data.

Growing community of inventors

North Salem, NY, United States of America

Frederick J Damerau