Method and system for multi-level artificial intelligence supercomputer design

Abstract

A system for enhancing large language models (LLMs) including an input component to receive an input text and split the input text into tokens, a batch processing component including one or more LLMs fine-tuned based on an aggregation of input data and that generate outputs responsive to receiving a token, a ranking component to score the LLM outputs and rank the tokens responsive to their respective scores, a clustering component to select a subset of highest-ranked tokens and consolidate the subset into refined context token batches, a control component to provide the refined context token batches for further training of the LLMs, and a query component to receive input queries, generate derived queries responsive to the input queries, transmit the derived queries to the batch processing component, receive responses from the batch processing component responsive to the derived queries, score and/or rank the responses, and transmit one or more responses.