Hierarchical and extensible language model inference

Abstract

At least one processor can receive an indication of processing to be performed in a natural language processing (NLP) machine learning (ML) pipeline, determine a first model class and a second model class for the processing. The first model class can be at a first hierarchical level of an ML hierarchy schema, and the second model class can be at a second hierarchical level of the ML hierarchy schema. The at least one processor can prepare a dictionary in a memory in communication with the at least one processor, which can comprise populating the dictionary with all required artifacts of the first model class and a subset of required artifacts of the second model class, wherein the second model class requires at least one of the required artifacts of the first model class. The at least one processor can perform NLP on text using the ML model and the dictionary.