Audiovisual secondary haptic signal reconstruction method based on cloud-edge collaboration

Abstract

An audio visual haptic signal reconstruction method includes first utilizing a large-scale audio-visual database stored in a central cloud to learn knowledge, and transferring same to an edge node; then combining, by means of the edge node, a received audio-visual signal with knowledge in the central cloud, and fully mining semantic correlation and consistency between modals; and finally fusing the semantic features of the obtained audio and video signals and inputting the semantic features to a haptic generation network, thereby realizing the reconstruction of the haptic signal. The method effectively solves the problems that the number of audio and video signals of a multi-modal dataset is insufficient, and semantic tags cannot be added to all the audio-visual signals in a training dataset by means of manual annotation. Also, the semantic association between heterogeneous data of different modals are better mined, and the heterogeneity gap between modals are eliminated.