Time-series image description method for dam defects based on local self-attention

Abstract

A time-series image description method for dam defects based on local self-attention mechanism is provided, including: performing frame sampling on an input time-series image of dam defect, extracting a feature sequence using a convolutional neural network and using the sequence as an input to a self-attention encoder, where the encoder includes a Transformer network based on a variable self-attention mechanism that dynamically establishes contextual feature relations for each frame; generating description text using a long short term memory (LSTM) network based on a local attention mechanism to enable each word predicted to be feature related to an image frame, improving text generation accuracy by establishing a contextual dependency between image and text. A dynamic mechanism is added to the present application for calculating the global self-attention of image frames, and LSTM networks with added local attention directly establish the correspondence between image and text modal data.