System and method for analyzing and tuning input dataset to a foundation model

Abstract

System and method for analyzing and tuning an input dataset to a foundation model is disclosed. The method includes, receiving an input dataset from a user, determining a textual representation of the received input dataset on an n-gram data level by processing the received input dataset, and determining a model score for the n-gram level by applying the determined textual representation to a trained artificial intelligence-based regression model. The method further includes, determining a best fit model and a unigram preparation data value for the n-gram level based on the determined model score, an n-gram preparation data value, and a type of n-gram using a trained classification model, wherein the best fit model is one of pretrained existing models and a dynamically trained n-gram model, predicting a bias score and an accuracy score for the input dataset by applying the determined textual representation and the input dataset onto the determined best fit model using a trained clustering model, generating a Prompt Inclusivity Index (PII) value based on the determined best fit model, the predicted bias score and the accuracy score, wherein the PII value indicates presence of bias data in the received input dataset, and outputting the generated PII value to a user via a user interface of a user device.