Framework and system for building assisted automations from recordings using program synthesis

Abstract

Described herein are systems and methods for automatically building automations from desktop recordings using program synthesis. The problem of building automations for desktop applications can be lifted to a generalized concept of automations that operate on worlds whose 'world state' can change asynchronously to the actions of the automation. Advantageously, in contrast with synthesis systems that take input-output demonstrations to synthesize a function that maps between them, the method presented here can synthesize from time-series traces of actions to automations that generalizes each step. The present disclosure describes ways to a) build assisted automations, b) synthesize them from recordings, c) running assisted automations using interpreters that observe the world state and adjust the actions accordingly, d) discovering them from an always-on recorder, e) suggesting them in partial progress from an always-on recorder, and lastly f) iteratively improving the assisted automation by recordings of every subsequent run.