Automated assistant control of external applications lacking automated assistant application programming interface functionality

Abstract

Implementations relate to an automated assistant that is capable of interacting with non-assistant applications that do not have functionality explicitly provided for interfacing with certain automated assistants. Application data, such as annotation data and/or GUI data, associated with a non-assistant application, can be processed to map such data into an embedding space. An assistant input command can then be processed and mapped to the same embedding space, and a distance from the assistant input command embedding and the non-assistant application data embedding can be determined. When the distance between the assistant input command embedding and the non-assistant application data embedding satisfies threshold(s), the automated assistant can generate instruction(s), for the non-assistant application, that correspond to the non-assistant application data. For instance, the instruction(s) can simulate user input(s) that cause the non-assistant application to perform one or more operations characterized by, or otherwise associated with, the non-assistant application data.