System and method of grounded large vision-language model for remote sensing

Abstract

A unified framework system and method for a computer implemented artificial intelligent assistant to perform multiple tasks for remote sensing, includes a task input field for receiving a task identity, a global image encoder configured to receive a remote-sensing image and a user query and encode patch-level tokens at a high resolution via interpolation positional encodings, an MLP adapter configured to receive the patch-level tokens and adapt the tokens to language space, and a large language model configured to generate natural language responses interleaved with corresponding object locations based on the language space and task specific prompts. The global image encoder further includes a region input field for receiving region location parameters. The framework system is configured to switch based on the input task identity between different types of remote sensing visual interpretation tasks.