Application control and text detection from application screen images

Abstract

Automation controls and associated text in images displayed by a computer application are automatically detected by way of region-based R-FCN and Faster R-CNN engines. Datasets comprising images containing application controls, where the application controls include images of application where width is greater than height, width is equal to height and height is greater than width are retrieved and each dataset is processed with the R-FCN and Faster R-CNN engines to generate a software robot configured to recognize corresponding application controls. Text is recognized by an optical character recognition system that employs a deep learning system trained to process a plurality of images to identify images representing text within each image and to convert the images representing text to textually encoded data. The deep learning system is trained with training data generated from a corpus of real-life text segments that are generated by a plurality of OCR modules.