Dictaphone Corporation.

John Dowd

Alwin B Carus

Larissa Lapshina

William F Cote

Jill Carrier

Alan Frankel

Ana Santisteban

Amy J Uhrbach

Bernardo Rechea

Kathryn Del La Femina

Wensheng(Vincent) Han

The present invention pertains to a system and method for the tokenization of text. The featurizer may be configured to receive input text and convert the input text into tokens. According to one aspect of the invention, the tokens may include only one type of character, the characters selected from the group consisting of letters, numbers, and punctuation. The tokenizer may also include a classifier. The classifier may be configured to receive the tokens from the featurizer. Furthermore, the classifier may be configured to analyze the tokens received from the featurizer to determine if the tokens may be input into a predetermined classification model using a preclassifier. If one of the tokens passes the preclassifier, then the token is classified using the predetermined classification model. Additionally, according to a first aspect of the invention, the tokenizer may also include a finalizer. The finalizer may be configured to receive the tokens and may be configured to produce a final output.

System and method for tokenization of text using classifier models

Growing community of inventors

Sudbury, MA, United States of America

John Dowd