Sri International

Michael A Cogswell

Ajay Divakaran

Karan Sikka

Yi Yao

Jesse Hostetler

Arijit Ray

Giedrius T Burachas

Yunye Gong

Pritish Sahu

Stephanie Nunn

Sara Rutherford-quach

Kamran Alipour

Yangyi Chen

In an example, a method for fine-tuning a Large Visual Language Model (LVLM) includes providing visual queries, each of the visual queries comprises at least an image and a textual query related to the image; processing, by the LVLM, the visual queries to extract visual embeddings from the visual queries, wherein the LVLM comprises a Visual Language Model (VLM), a first Large Language Model (LLM), and a linear projection layer interconnecting the VLM and the LLM; for visual queries: i) generating, by the LVLM, a response to the corresponding visual query based on the corresponding visual embedding; ii) evaluating, by a second LLM, the generated response to verify that the generated response satisfies predefined criteria; and iii) providing, by the second LLM, a feedback to the LVLM, in response to the evaluating the generated response; and fine-tuning the LVLM using aggregated feedback provided by the second LLM for the visual queries.

Instruction-guided visual embeddings and feedback-based learning in large vision-language models

A computing system comprising a memory configured to store an artificial intelligence (AI) model and an image, and a computation engine executing one or more processors may be configured to perform the techniques for error-based explanations for AI behavior. The computation engine may execute the AI model to analyze the image to output a result. The AI model may, when analyzing the image to output the result, process, based on data indicative of the result, the image to assign an error score to each image feature extracted from the image, and obtain, based on the error scores, an error map. The AI model may next update, based on the error map and to obtain a first updated image, the image to visually indicate the error score assigned to each of the image features, and output one or more of the error scores, the error map, and the first updated image.

Error-based explanations for artificial intelligence behavior

A method, apparatus and system for training an embedding space for content comprehension and response includes, for each layer of a hierarchical taxonomy having at least two layers including respective words resulting in layers of varying complexity, determining a set of words associated with a layer of the hierarchical taxonomy, determining a question answer pair based on a question generated using at least one word of the set of words and at least one content domain, determining a vector representation for the generated question and for content related to the at least one content domain of the question answer pair, and embedding the question vector representation and the content vector representations into a common embedding space where vector representations that are related, are closer in the embedding space than unrelated embedded vector representations. Requests for content can then be fulfilled using the trained, common embedding space.

System and method for content comprehension and response

Growing community of inventors

West Windsor, NJ, United States of America

Michael A Cogswell