Automatically identifying and interacting with hierarchically arranged elements

Abstract

A computer-implemented method of managing hierarchically arranged elements is disclosed. The method comprising: receiving, by a processor, a digital image of an electronic form having at least one group of elements, including at least one field that is programmed to receive input data; detecting, by the processor, a set of objects represented in the digital image; identifying, for each object of the set of objects, values for multiple attributes, including a type and a position, a value of the type being a group label corresponding to a group label in the electronic form, a field label corresponding to a field label in the electronic form, or a field corresponding to a field in the electronic form; building a set of feature vectors, including a feature vector for each pair of objects in the set of objects such that a first object of the pair has a type of a field and a second object of the pair has a type of a group label or a field label, the feature vector including a first feature for the type of each of the pair of objects and a second feature for the position of each of the pair of objects; determining, for each object of the set of objects having a type of a field, an associated group label and an associated field label based on the set of feature vectors; searching, for each of at least one the set of objects having a type of a field, a database for field data for the object based on the associated group label and the associated field label; causing, by the processor, displaying each of the at least one objects having a type of a field in association with the corresponding field data, thereby automatically completing the electronic form.