International Business Machines Corporation

Richard G Carlgren

Walter S Rosenbaum

Daniel John McCloskey

Brian O'Donovan

Alan Richard Tannenbaum

Pavel Volkov

William David Modlin

Alexei Nevidomski

Mayo Takeuchi

Martin A Reed

Alexandre Troussov

A method and system for language identification are provided. The system includes a feature set of a plurality of character strings of varying length with associated information. The associated information includes one or more significance scores for a character string for one or more of a plurality of languages. Means are provided for detecting character strings from the feature set within a token from an input text. The system uses a finite-state device and the associated information is provided as glosses at the final nodes of the finite-state device for each character string. The associated information can also include significance scores based on linguistic rules.

Method and system for language identification

A data processing method is disclosed for storing and retrieving text. The storage part of the method includes the steps of compiling a vocabulary list of words occurring in the text and augmenting the vocabulary list with lemmas of the words in the text, as an augmented word list. It then continues with the steps of compiling a cross reference table relating the lemmas of the words to locations of the words in the text and storing the text, the augmented word list and the cross reference table. The retrieval part of the method includes the steps of inputting a query word to access a portion of the stored text, searching the augmented vocabulary list using the query word as a search term, and accessing the cross reference table with a lemma of the query word to locate the portion of the stored text. The resulting invention enables a faster performance for 'fuzzy' searches of text in documents, while enabling the cross reference lists used in the retrieval process, to be compactly stored.

Method for use of morphological information to cross reference keywords

The combination of dictionary driven hyphenation, specialized algorithmic hyphenation and intelligent blank insertion provides improved right margin justification capability in a text processing system. When hyphenation is required for right margin justification, the system compares the word to be hyphenated to a prestored dictionary of words containing hyphenation points. When the word to be hyphenated matches one of the dictionary words the hyphenation points are retrieved and the word is split at the right margin. If the word to be hyphenated does not match one of the dictionary words, then a specialized list of prestored hyphenated suffixes and prestored statistical character digrams are compared to the word to determine the appropriate hyphenation points. Once the word has been split, the system searches the line for sets of predetermined words which may be separated from other words in the sentence by adding space to the line with a minimum of aesthetic distortion. Space is then added to the line until the line ending equals the right margin. The text is then printed.

Mixed mode enhanced resolution hyphenation function for a text

A system for proofreading a text document and automatically detecting and replacing text words in the document which exceed a predetermined understandability level for the documents intended audience. Text words and synonyms are stored in a dictionary which includes an understandability code for each word based statistically on textbook grade levels. The operator enters a grade level code into the system for the intended document audience. The system scans the document for words which exceed the desired grade level, highlights those words on the system display and prompts the operator with synonyms which can be used to replace the highlighted word. The operator may select a desired replacement synonym by placing the system cursor underneath the word and depressing and enter key from the system keyboard.

Automatic text grade level analyzer for a text processing system

Growing community of inventors

Gaithersburg, MD, United States of America

Richard G Carlgren