International Business Machines Corporation

Jerome R Bellegarda

David Nahamoo

Krishna Sundaram Nathan

Michael Alan Picheny

Lalit R Bahl

Eveline J Bellegarda

Dimitri Kanevsky

Ponani S Gopalakrishnan

Peter V De Souza

Edward Adam Epstein

John M Lucassen

Arthur J Nadas

Peter V Desouza

A method of, and apparatus for, operating an automatic message recognition system. In accordance with the method the following steps are executed: a user's speech is converted to a first signal; a user's handwriting is converted to a second signal; and the first signal and the second signal are processed to decode a consistent message, conveyed separately by the first signal and by the second signal, or conveyed jointly by the first signal and the second signal. The step of processing includes the steps of converting the first signal into a plurality of first multi-dimensional vectors and converting the second signal into a plurality of second multi-dimensional vectors. For a system employing a combined use of speech and handwriting the step of processing includes a further step of combining individual ones of the plurality of first multi-dimensional vectors and individual ones of the plurality of second multi-dimensional vectors to form a plurality of third multi-dimensional vectors. The multi-dimensional vectors are employed to train a single set of word models, for joint use of speech and handwriting, or two sets of word models, for sequentially employed or merged speech and handwriting.

Message recognition employing integrated speech and handwriting information

An automatic handwriting recognition system wherein each written (chirographic) manifestation of each character is represented by a statistical model (called a hidden Markov model). The system implements a method which entails sampling a pool of independent writers and deriving a hidden Markov model for each particular character (allograph) which is independent of a particular writer. The HMMs are used to derive a chirographic label alphabet which is independent of each writer. This is accomplished during what is described as the training phase of the system. The alphabet is constructed using supervised techniques. That is, the alphabet is constructed using information learned in the training phase to adjust the result according to a statistical algorithm (such as a Viterbi alignment) to arrive at a cost efficient recognition tool. Once such an alphabet is constructed a new set of HMMs can be defined which more accurately reflects parameter typing across writers. The system recognizes handwriting by applying an efficient hierarchical decoding strategy which employs a fast match and a detailed match function, thereby making the recognition cost effective.

System and method for automatic handwriting recognition with a

A computer-based system and method for recognizing handwriting. The present invention includes a pre-processor, a front end, and a modeling component. The present invention operates as follows. First, the present invention identifies the lexemes for all characters of interest. Second, the present invention performs a training phase in order to generate a hidden Markov model for each of the lexemes. Third, the present invention performs a decoding phase to recognize handwritten text. Hidden Markov models for lexemes are produced during the training phase. The present invention performs the decoding phase as follows. The present invention receives test characters to be decoded (that is, to be recognized). The present invention generates sequences of feature vectors for the test characters by mapping in chirographic space. For each of the test characters, the present invention computes probabilities that the test character can be generated by the hidden Markov models. The present invention decodes the test character as the recognized character associated with the hidden Markov model having the greatest probability.

Continuous parameter hidden Markov model approach to automatic

A general approach is provided for the combined use of several sources of information in the automatic recognition of a consistent message. For each message unit (e.g., word) the total likelihood score is assumed to be the weighted sum of the likelihood scores resulting from the separate evaluation of each information source. Emphasis is placed on the estimation of weighing factors used in forming this total likelihood. This method can be applied, for example, to the decoding of a consistent message using both handwriting and speech recognition. The present invention includes three procedures which provide the optimal weighing coefficients.

Computer program product for automatic recognition of a consistent

Methods and apparatus are disclosed for recognizing handwritten characters in response to an input signal from a handwriting transducer. A feature extraction and reduction procedure is disclosed that relies on static or shape information, wherein the temporal order in which points are captured by an electronic tablet may be disregarded. A method of the invention generates and processes the tablet data with three independent sets of feature vectors which encode the shape information of the input character information. These feature vectors include horizontal (x-axis) and vertical (y-axis) slices of a bit-mapped image of the input character data, and an additional feature vector to encode an absolute y-axis displacement from a baseline of the bit-mapped image. It is shown that the recognition errors that result from the spatial or static processing are quite different from those resulting from temporal or dynamic processing. Furthermore, it is shown that these differences complement one another. As a result, a combination of these two sources of feature vector information provides a substantial reduction in an overall recognition error rate. Methods to combine probability scores from dynamic and the static character models are also disclosed.

Automatic handwriting recognition using both static and dynamic

A computer-based system and method for recognizing handwriting. The present invention includes a preprocessor, a front end, and a modeling component. The present invention operates as follows. First, the present invention identifies the lexemes for all characters of interest. Second, the present invention performs a training phase in order to generate a hidden Markov model for each of the lexemes. Third, the present invention performs a decoding phase to recognize handwritten text. Hidden Markov models for lexemes are produced during the training phase. The present invention performs the decoding phase as follows. The present invention receives test characters to be decoded (that is, to be recognized). The present invention generates sequences of feature vectors for the test characters by mapping in chirographic space. For each of the test characters, the present invention computes probabilities that the test character can be generated by the hidden Markov models. The present invention decodes the test character as the recognized character associated with the hidden Markov model having the greatest probability.

Automatic recognition of a consistent message using multiple

Method and apparatus for automatic recognition of handwritten text based on a suitable representation of handwriting in one or several feature vector spaces(s), Gaussian modeling in each space, and mixture decoding to take into account the contribution of all relevant prototypes in all spaces. The feature vector space(s) is selected to encompass both a local and a global description of each appropriate point on a pen trajectory. Windowing is performed to capture broad trends in the handwriting, after which a linear transformation is applied to suitably eliminate redundancy. The resulting feature vector space(s) is called chirographic space(s). Gaussian modeling is performed to isolate adequate chirographic prototype distributions in each space, and the mixture coefficients weighting these distributions are trained using a maximum likelihood framework. Decoding can be performed simply and effectively by accumulating the contribution of all relevant prototype distributions. Post-processing using a language model may be included.

Statistical mixture approach to automatic handwriting recognition

In speech recognition and speech coding, the values of at least two features of an utterance are measured during a series of time intervals to produce a series of feature vector signals. A plurality of single-dimension prototype vector signals having only one parameter value are stored. At least two single-dimension prototype vector signals having parameter values representing first feature values, and at least two other single-dimension prototype vector signals have parameter values representing second feature values. A plurality of compound-dimension prototype vector signals have unique identification values and comprise one first-dimension and one second-dimension prototype vector signal. At least two compound-dimension prototype vector signals comprise the same first-dimension prototype vector signal. The feature values of each feature vector signal are compared to the parameter values of the compound-dimension prototype vector signals to obtain prototype match scores. The identification values of the compound-dimension prototype vector signals having the best prototype match scores for the feature vectors signals are output as a sequence of coded representations of an utterance to be recognized. A match score, comprising an estimate of the closeness of a match between a speech unit and the sequence of coded representations of the utterance, is generated for each of a plurality of speech units. At least one speech subunit, of one or more best candidate speech units having the best match scores, is displayed.

Speech coding apparatus with single-dimension acoustic prototypes for a

A speech coding apparatus and method for use in a speech recognition apparatus and method. The value of at least one feature of an utterance is measured during each of a series of successive time intervals to produce a series of feature vector signals representing the feature values. A plurality of prototype vector signals, each having at least one parameter value and a unique identification value are stored. The closeness of the feature vector signal is compared to the parameter values of the prototype vector signals to obtain prototype match scores for the feature value signal and each prototype vector signal. The identification value of the prototype vector signal having the best prototype match score is output as a coded representation signal of the feature vector signal. Speaker-dependent prototype vector signals are generated from both synthesized training vector signals and measured training vector signals. The synthesized training vector signals are transformed reference feature vector signals representing the values of features of one or more utterances of one or more speakers in a reference set of speakers. The measured training feature vector signals represent the values of features of one or more utterances of a new speaker/user not in the reference set.

Speech coding apparatus having speaker dependent prototypes generated

An apparatus for generating a set of acoustic prototype signals for encoding speech includes a memory for storing a training script model comprising a series of word-segment models. Each word-segment model comprises a series of elementary models. An acoustic measure is provided for measuring the value of at least one feature of an utterance of the training script during each of a series of time intervals to produce a series of feature vector signals representing the feature values of the utterance. An acoustic matcher is provided for estimating at least one path through the training script model which would produce the entire series of measured feature vector signals. From the estimated path, the elementary model in the training script model which would produce each feature vector signal is estimated. The apparatus further comprises a cluster processor for clustering the feature vector signals into a plurality of clusters. Each feature vector signal in a cluster corresponds to a single elementary model in a single location in a single word-segment model. Each cluster signal has a cluster value equal to an average of the feature values of all feature vectors in the signal. Finally, the apparatus includes a memory for storing a plurality of prototype vector signals. Each prototype vector signal corresponds to an elementary model, has an identifier, and comprises at least two partition values. The partition values are equal to combinations of the cluster values of one or more cluster signals corresponding to the elementary model.

Growing community of inventors

Goldens Bridge, NY, United States of America

Jerome R Bellegarda