Non-lexicalized features for language identity classification using subword tokenization

Abstract

A natural language identity classifier system is described, which employs a supervised machine learning (ML) model to perform language identity classification on input text. The ML model takes, as input, non-lexicalized features of target text derived from subword tokenization of the text. Specifically, these non-lexicalized features are generated based on statistics determined for tokens identified for the input text. According to an embodiment, at least some of the non-lexicalized features are based on natural language-specific summary statistics that indicate how often tokens were found within a corpus for each natural language. Use of such summary statistics allows for generation of natural language specific conditional probability-based features. Because of the inherent interpretability of a trained non-lexicalized ML model as described herein, it is possible to modify behavior of the trained ML model by adjusting summary statistics maintained for natural language tokens and/or by adjusting data for the subword tokenizers.