Sap Se

Mithilesh Kumar Singh

Satyadeep Dey

Warren Mark Fernandes

Sai Phani Sharath Chandra Danthalapelli

Sudha Karanam Narasimha Murthy

Sharmika Parmar

Chandrakanth S

Sonam Saxena

Suvajit Dutta

Vinay Kumar

Mohammad Saad Rashid

Raghavendra D

Philippe Long

Deepak Pandian

Gagan K

Samir Patil

Anubhav Sadana

Provided are systems and methods for batching instructions of a bot during execution/runtime of the bot. The bot may be a software program that is designed to perform user interface interactions (e.g., button clicks, opening/closing pages, text entry, etc.) in place of a human. In one example, a method may include receiving a request to execute a bot program configured to perform a sequence of actions on a user interface of a software application, identifying a plurality of actions of the bot program that can be batched, assembling a plurality of instructions for performing the plurality of actions of the bot program into a batched payload, and transmitting an automation request with the batched payload to a back-end of the application on a server.

Batch processing during bot runtime

Provided are systems and methods for automatically detecting a change in screen content and generating a hint message in response. As another example, the system may auto-capture the detected change in the screen content instead of or in addition to the hint message. In one example, a method may include capturing user interface metadata of content being displayed by a client-side of the software application, activating a recorder that is configured to record events transmitted between the client-side and a server-side of the software application, receiving updated user interface metadata based on a user interaction on the client-side of the software application, determining that user interface content has changed based on a comparison of the captured user interface metadata to the updated user interface metadata, and displaying a hint message via the user interface of the client-side of the software application.

Screen capture hint and automated screen capture

A data annotation server accesses a request from a machine learning server for annotated images of a user interface containing a specified user interface element. The data annotation server programmatically determines whether user interfaces generated by an application server include the specified user interface element. If so, an image of the user interface is stored and a location or bounding box of the user interface element is determined. The stored image of the user interface is annotated with the determined location of the user interface element. The image and the annotation are provided to the machine learning server, which uses the images and annotations to train a machine learning model.

Data generation and annotation for machine learning

Disclosed herein are system, method, and computer program product embodiments for surface automation in black box environments. An embodiment operates by determining scenarios of an application for automation; detecting the scenario during an execution of an application; capturing and storing one or more user interface screenshots of the scenario; identifying and storing user interface information from the user interface screenshot; implementing a sequential set of instructions comprising at least one textual element detection technique and at least one non-textual element detection technique; and executing the sequential set of instructions.

Surface automation in black box environments

Provided are systems and methods for converting user interface events that occur in a software application developed via a WebGUI framework into activity descriptions and into a bot software program. In one example, a method may include recording events transmitted between a user interface of a web application open within a web browser on a client device and a back-end of the web application on a server, identifying codes associated with user interface elements which are assigned to the recorded events, converting the identified codes assigned to the recorded events into a human-readable descriptions of the recorded events based on a predefined mapping between the codes and the human-readable descriptions stored in the predefined mapping, and displaying the human-readable descriptions of the recorded events based on the identified order among the recorded events.

Conversion of user interface events

A computer-implemented method can receive a request from a robotic process automation engine to identify a target user interface control element in a webpage represented by a current master data frame. The current master data frame comprises a current document object model (DOM). The method can determine that a target user interface control element identifier associated with the target user interface control element is absent in the current DOM. The method can retrieve an archived version of the target user interface control element from an archived master data frame of the webpage. The method can find an equivalent user interface control element within the current master data frame based at least on the archived version of the target user interface control element, and output an equivalent user interface control element identifier associated with the equivalent user interface control element.

Adaptive web-based robotic process automation

The present disclosure involves systems, software, and computer implemented methods for automatically detecting user interface elements. One example method includes accessing master frame information for a user interface of an application. The master frame information includes first captured user interface information captured during a first execution of a scenario for the application. A subsequent execution of the scenario is performed, including capturing second captured user interface information. A determination is made that the subsequent execution of the scenario has not passed successfully, due to a non-functional error. A determination is made that the non-functional error is based on a failure to locate a user interface element specified in the master frame. A recovery strategy is performed, using the first captured user interface information and the second captured user interface information, to automatically locate the user interface element.

Automatic detection of user interface elements

In one aspect, there is provided a method for software testing. In one aspect, there is provided a method. The method may include executing a test script including at least one test instruction requiring an input at a user interface element displayed on a screen of a device under test; determining, based on a machine learning model, a candidate location on the screen of the device under test, the candidate location representing a candidate portion of the screen having the user interface element for the required input associated with the at least one test instruction; recognizing, based on optical character recognition, one or more characters in the determined candidate location; selecting, based on the recognized characters, the determined candidate location as the user interface element having the required input; and executing an inserted value at the determined candidate location to test a result of the test script execution.

Growing community of inventors

Ara, India

Mithilesh Kumar Singh