Citibank, N.a.

Mariusz Saternus

Daniel Lewandowski

Philip Davies

Stuart Murray

Biraj Krushna Rath

Payal Jain

Tariq Husayn Maonah

Shardul Malviya

Samantha Cory

Deepak Jain

Wayne Liao

Chamindra Desilva

Vishal Mysore

James Myers

Ramkumar Ayyadurai

Sofia Rahman

Prashant Praveen

Christopher Tucker

Miriam Silver

Prithvi Narayana Rao

William Franklin Cameron

Pramod Goyal

Manjit Rajaretnam

Zheyu Wang

Sourabh Deb

Haolin Jin

Julisia Jackson

Nigil Satish Jeyashekar

Systems and methods for restructuring prompts in order to improve accuracy of outputs from models are disclosed herein. The system receives a user prompt indicating a request for data. The system generates a first and second output using a model, the first output generated based on the user prompt and the second output generated based on pseudocode. The system compares the first and second outputs to determine a match accuracy between the two outputs. If the two outputs sufficiently match, the system approves the user prompt. If the two outputs do not sufficiently match, the system initiates a prompt restructuring process, whereby the user prompt is restructured using pseudocode to improve the accuracy of the first output. The process is repeated iteratively until the restructured user prompt generates a first output that sufficiently matches the second output generated based on the pseudocode.

Prompt refinement to improve accuracy of outputs from machine learning models

The systems and methods disclosed herein receives, from a computing device, operational data indicating software or hardware assets used on informational assets, and obtains a set of alphanumeric characters defining operative boundaries for expected system assets, which include a set of common attributes. Using the set of attributes, a first set of AI models determines observed system assets from the operational data, each with specific features. A second set of AI models associates each information asset with the corresponding observed system assets. For each observed system asset, a third set of AI models identifies criteria within the alphanumeric characters, compares the criteria with the asset's features to identify gaps, and generates actions to ensure the observed system asset meets the identified criteria.

Managing operational resilience of system assets using an artificial intelligence model

The systems and methods disclosed herein receive an output generation request from that includes input for generating an output using a language model. The input includes a set of alphanumeric characters associated with operative standards for a first set of actions. The system divides the set of alphanumeric characters into text subsets. For each text subset, a vector representation is determined. Prompts are created for each vector representation including the set of alphanumeric characters, query contexts, keywords, and/or the text subset. Each vector representation's prompt is input into the language model, which generates a second set of actions of related actions, where subsequently generated actions are based on prior generated actions. The system aggregates the second set of actions into a third set of actions and displays a graphical layout. The graphical layout displays a representation of the set of alphanumeric characters and the corresponding actions.

Identifying and analyzing actions from vector representations of alphanumeric characters using a large language model

The systems and methods disclosed herein receive alphanumeric characters defining operative boundaries for expected model use cases, along with operational data. The expected model use cases share common attributes, which are used by a first AI model to construct observed model use cases from the operational data. Each observed model use case includes features such as a text-based description, expected input and output,? AI model(s) generating the expected output from the input, and/or data supporting the AI models. For each observed model use case, a second AI model maps the alphanumeric characters and features to a risk category, selecting from multiple risk categories based on the level of risk associated with the features. The system identifies criteria for the observed model use case within the alphanumeric characters and generates gaps by comparing the criteria with the features of the observed model use case.

Identifying and remediating gaps in artificial intelligence use cases using a generative artificial intelligence model

The systems and methods disclosed herein obtain a set of alphanumeric characters defining constraints for agents and the agents' operational data. Each agent uses an output from a first set of artificial intelligence (AI) models and predefined objectives to autonomously generate proposed actions for execution on software application(s). For each agent, a second set of AI models evaluates the agent by identifying gaps in the proposed actions by comparing them with the expected actions. Using a third set of AI models and the identified gaps, the systems modify the proposed actions by adding, altering, or removing actions from the proposed actions.

Validating autonomous artificial intelligence (AI) agents using generative AI

The systems and methods disclosed herein receive alphanumeric characters defining operative boundaries for expected model use cases, along with operational data. The expected model use cases share common attributes, which are used by a first AI model to construct observed model use cases from the operational data. Each observed model use case includes features such as a text-based description, expected input and output, AI model(s) generating the expected output from the input, and/or data supporting the AI models. For each observed model use case, a second AI model maps the alphanumeric characters and features to a risk category, selecting from multiple risk categories based on the level of risk associated with the features. The system identifies criteria for the observed model use case within the alphanumeric characters and generates gaps by comparing the criteria with the features of the observed model use case.

The systems and methods disclosed herein relate to querying data using artificial intelligence models. A generalized model receives an output generation request and partitions it into segments mapped to specific domains, where each domain indicates associated databases and guidelines. The segments are routed to domain-specific models trained on domain-specific data, which generate query fragments by comparing performance metrics and system resource usage metrics. The query fragments are aggregated into an overall query that satisfies guidelines across domains. The systems and methods can include a feedback loop to adjust the domain-specific models using user interactions and performance metrics to dynamically adapt to a skill level or experience of the user.

Querying data using specialized and generalized artificial intelligence models

The systems and methods disclosed herein receives artifacts generated using a first set of models within a multi-model superstructure. The multi-model superstructure includes a second set of models to test the first set of models. The multi-model superstructure dynamically routes the artifacts of the first set of models to one or more models of the second set of models by (i) determining a set of dimensions of the artifacts against which to evaluate the artifacts and (ii) identifying the models in the second set used to test the particular dimension. The second set of models then assesses each artifact against a set of assessment metrics. If an artifact fails to meet one or more assessment metrics, the second set of models generates actions to align the artifact with the set of assessment metrics.

Dynamic multi-model monitoring and validation for artificial intelligence models

The systems and methods disclosed herein enable mapping of gaps in controls to operative standards. The system receives an output generation request using an artificial intelligence (AI) model, where the input includes a set of gaps associated with one or more scenarios failing to satisfy the operative standards of a set of vector representations. Each gap in the set of gaps includes attributes defining the scenario. Using the received input, the system constructs prompts for each gap, where the prompts include information related to the scenario and/or the operative standards. Each prompt compares the corresponding gap against the operative standards or the set of vector representations. For each gap, the system maps the gap to the operative standards by supplying the prompt of the particular gap into the AI model and, in response, receiving from the AI model the operative standards associated with the particular gap.

Mapping identified gaps in controls to operative standards using a generative artificial intelligence model

The systems and methods disclosed herein relate to a model validation platform that enables dynamic validation of a user's prompt for a large language model (LLM) in order to evaluate the validity of the prompt and the suitability of a large language model for processing the prompt. For example, the platform enables an estimation of the resource allocation associated with processing the prompt with a given LLM, as well as a modification of the prompt, prior to the processing the prompt with the selected LLM. The platform can further validate the output prior to transmitting the output to a server system for display to the user. By doing so, the platform enables dynamic evaluation of a request to execute an LLM, as well as evaluation of resulting outputs, for accuracy and efficiency improvements in data processing or software development pipelines.

Dynamic evaluation of language model prompts for model selection and output validation and methods and systems of the same

The systems and methods disclosed herein enable evaluation of machine learning model outputs within a virtual environment. The disclosed model validation platform enables testing of code generated for detection of malicious or anomalous outputs. For example, the model validation platform can construct a virtual machine isolated from the system and test model-generated code for validation of LLM-generated outputs. In some implementations, the model validation platform determines parameters of the virtual machine and/or associated validation test based on an evaluation of the machine learning model's output and/or the associated underlying prompt. For example, the parameters of the validation test depend on an evaluation of the user or the provided input (e.g., depending on the presence of sensitive data within the prompt). By doing so, the system enables dynamic evaluation of machine learning model outputs to improve the security and robustness of associated generated code.

Dynamic input-sensitive validation of machine learning model outputs and methods and systems of the same

The disclosed data generation platform enables selection of particular machine learning models on the basis of a predicted resource allocation requirement associated with a given prompt. For example, the model validation platform can evaluate the resource use (e.g., cost) associated with processing a user's prompt with a given type of model. Based on this estimated resource use, the model validation platform can route the prompt to a suitable model to optimize a performance metric value, thereby improving the efficiency of the system. In some implementations, the data generation platform trains a model to accurately estimate resource usage based on ground-truth model-related costs, thereby improving the effectiveness of model selection for efficiency improvements.

Growing community of inventors

Cracow, Poland

Mariusz Saternus