Method and system for extracting information based on user interactions performed on a screen of a computing system

Abstract

The present disclosure relates to method and system for extracting information based on user interactions performed on screen of a computing system. Method includes receiving processed input data comprising video capturing screen and user interactions events occurring from user interactions, and co-ordinate information. Thereafter, computing system determines Regions of Interest (RoI) on screen based on captured user interactions, events and co-ordinate information, and identifies type of user interaction performed with at least one screen element in the RoI, as either keyboard or mouse type interaction. Subsequently, computing system performs one of: determining type of screen element as a text box or a table based on pattern recognition when type of user interaction is identified as keyboard type interaction or determining type of screen element to be selectable User Interface elements and extracting content and label of selectable UI elements, when interaction is identified as mouse type interaction.