Training a tokenizer using altered text data

Abstract

Techniques are disclosed for altering tokenizer training data causing a tokenizer to generate an improved library of tokens from the altered training data for tokenizing new source data during training of a machine learning model e.g., for natural language processing. A system retrieves, from a source database, a set of original text data that includes characters. The server system identifies a plurality of strings included in the set of original text data. The system alters strings included in the original text data to generate altered text data by selecting current characters included in the strings to be altered. The system trains, using the altered text data, a tokenizer by assigning a threshold to the tokenizer and inputting the altered text data into the tokenizer, where the tokenizer generates tokens for strings in the altered text data based on the threshold. The system stores the trained tokenizer.