Explainable multi-entity event recognition

Abstract

An image processing system has a memory storing a video depicting a multi-entity event, a trained reinforcement learning policy and a plurality of domain specific language functions. A graph formation module computes a representation of the video as a graph of nodes connected by edges. A trained machine learning system recognizes entities depicted in the video and recognizes attributes of the entities. Labels are added to the nodes of the graph according to the recognized entities and attributes. The trained machine learning system computes a predicted multi-entity event depicted in the video. For individual ones of the edges of the graph, select a domain specific language function from the plurality of domain specific language functions and assign it to the edge, the selection being made at least according to the reinforcement learning policy. An explanation is formed from the domain specific language functions.