Intuit, Inc.

Kamalika Das

Jiaxin Zhang

Sricharan Kallur Palli Kumar

Tharathorn Joy Rimchala

Lalla Mouatadid

Xiang Gao

Sricharan Kumar

Avinash Baidya

Shuyi Li

Apoorva Banubakode

At least one processor may receive conversation data comprising a user input and a response to the user input generated by a large language model (LLM). The at least one processor may detect a behavior gap between the conversation data and at least one test conversation data comprising human-to-human communication samples. Based on the behavior gap, the at least one processor may retrain the LLM and/or provide an alternative response workflow for responding to the user input.

Large language model performance evaluation and modification

Systems and methods are disclosed for detecting hallucinations in large language models (LLMs). An example method includes receiving a first prompt for submission to the first LLM, generating, using the first LLM, a plurality of semantically equivalent prompts to the first prompt, generating, using the first LLM, a first response to the first prompt and a plurality of second responses to the plurality of semantically equivalent prompts, generating, using a second LLM, a plurality of third responses to the semantically equivalent prompts, generating a semantic consistency score for the first response based at least in part on the first prompt, the plurality of semantically equivalent prompts, the plurality of second responses, and the plurality of third responses, and determining whether or not the first response is an accurate response to the first prompt based at least in part on the semantic consistency score.

Semantic aware hallucination detection for large language models

Augmented Denoising Diffusion Implicit Models ('DDIMs') using a latent trajectory optimization process can be used for image generation and manipulation using text input and one or more source images to create an output image. Noise bias and textual bias inherent in the model representing the image and text input is corrected by correcting trajectories previously determined by the model at each step of a diffusion inversion process by iterating multiple starts the trajectories to find determine augmented trajectories that minimizes loss at each step. The trajectories can be used to determine an augmented noise vector, enabling use of an augmented DDIM and resulting in more accurate, stable, and responsive text-based image manipulation.

Augmented diffusion inversion using latent trajectory optimization

Certain aspects of the disclosure provide a method, comprising: processing input data with an ensemble of nonlinear machine learning models; generating a sparse high-dimensional embedding based on one or more leaf nodes of each nonlinear machine learning model in the ensemble of nonlinear machine learning models; projecting the high-dimensional embedding into a lower-dimensional embedding, wherein the lower-dimensional embedding is less sparse than the high-dimensional embedding; processing the lower-dimensional embedding with a linear machine learning model to generate a binary class prediction; determining a confidence for the binary class prediction; and outputting: the binary class prediction if the confidence is greater than or equal to a threshold; or a flipped binary class prediction if the confidence is lower than the threshold.

Uncertainty quantification for machine learning classification modelling

A method includes receiving a user input including natural language text. The method also includes generating modified inputs from the user input. The method also includes executing a machine learning model on the modified inputs to generate model outputs. The method also includes sampling the model outputs using a statistical sampling strategy to generate sampled model outputs. The method also includes clustering the sampled model outputs into clusters. The method also includes generating a confidence metric of the clusters. The method also includes routing, automatically in a computing system, the user input based on whether the confidence metric satisfies a threshold value.

Use of semantic confidence metrics for uncertainty estimation in large language models

Systems and methods are disclosed for model based document image enhancement. Instead of requiring paired dirty and clean images for training a model to clean document images (which may cause privacy concerns), two models are trained on the unpaired images such that only the dirty images are accessed or only the clean images are accessed at one time. One model is a first implicit model to translate the dirty images from a source space to a latent space, and the other model is a second implicit model to translate the images from the latent space to clean images in a target space. The second implicit model is trained based on translating electronic document images in the target space to the latent space. In some implementations, the implicit models are diffusion models, such as denoising diffusion implicit models based on solving ordinary differential equations.

Model based document image enhancement

Disambiguity in large language models (LLMs) includes receiving an original query in a user interface, generating an ambiguity query from the original query, and sending, via an application programming interface (API) of an LLM, the ambiguity query to the LLM. The ambiguity query includes the original query and training the LLM to recognize ambiguities. The method further includes receiving, via the API and responsive to the ambiguity query, a binary response and detecting, based at least in part on the binary response, the original query as ambiguous. Disambiguity may include detecting an ambiguity location in the original query using perturbed queries and the LLM.

Disambiguity in large language models

A method including receiving a user input for input to a language processing machine learning model (MLM). The method also includes generating modified inputs that are based on, and semantically related to, the user input. The method also includes executing the MLM to generate model outputs of the MLM. The MLM takes as input instances of each of the modified inputs. The method also includes sampling the model outputs using a statistical sampling strategy to generate sampled model outputs. The method also includes clustering the sampled model outputs into clusters. Each cluster of the clusters represents a distinct semantic meaning of the sampled model outputs. The method also includes generating a confidence metric for the user input. The confidence metric includes a predictive entropy of the clusters. The method also includes routing the user input based on whether the confidence metric satisfies or fails to satisfy a threshold value.

Systems and methods are disclosed for classifying objects by a machine learning (ML) model. The ML model includes one or more layer level classification models to generate classifications and uncertainty metrics in the classifications and a meta-model to generate a final classification and confidence based on the underlying classifications and uncertainty metrics. In some implementations, the ML model provides an object to be classified to one or more layer level classification models, and the layer level classification models generate a classification for the object and an uncertainty metric in the classification. The meta-model receives the classifications and uncertainty metrics from the one or more layer level classification models and generates the final classification and confidence in the final classification. The uncertainty metrics may also be output by the ML model or used to adjust the meta-model to improve the final classification and confidence.

Growing community of inventors

Saratoga, CA, United States of America

Kamalika Das