Uipath Inc.

Mircea Neagovici

Stefan A Adam

Dragos H Bobolea

Virgil Tudor

Horia Cristescu

Graphical elements in a user interface (UI) may be detected in robotic process automation (RPA) using convolutional neural networks (CNNs). Such processes may be particularly well-suited for detecting graphical elements that are too small to be detected using conventional techniques. The accuracy of detecting graphical elements (e.g., control objects) may be enhanced by providing neural network-based processing that is robust to changes in various UI factors, such as different resolutions, different operating system (OS) scaling factors, different dots-per-inch (DPI) settings, and changes due to UI customization of applications and websites, for example.

Detecting user interface elements in robotic process automation using convolutional neural networks

Described systems and methods allow the automatic extraction of structured information from images of structured text documents such as invoices and receipts. Some embodiments employ optical character recognition (OCR) technology to extract individual text tokens (e.g., words) and token bounding boxes from a document image. A feature vector of each text token comprises a first part determined according to a character content of the text token, and a second part determined according to an image content of the token's bounding box. A neural network classifier produces a label indicative of a type of information (e.g. 'billing address', 'due date', etc.) carried by each text token. In some embodiments, documents are linearized by ordering text tokens in a sequence according to a reading order of a natural language (e.g., English, Arabic) in which the respective document is formulated. Token feature vectors are fed to the classifier in the order indicated by the token sequence.

Growing community of inventors

Bellevue, WA, United States of America

Mircea Neagovici