Systems and methods for time series analysis using attention models

Abstract

A system for time series analysis using attention models is disclosed. The system may capture dependencies across different variables through input embedding and may map the order of a sample appearance to a randomized lookup table via positional encoding. The system may capture capturing dependencies within a single sequence through a self-attention mechanism and determine a range of dependency to consider for each position being analyzed. The system may obtain an attention weighting to other positions in the sequence through computation of an inner product and utilize the attention weighting to acquire a vector representation for a position and mask the sequence to enable causality. The system may employ a dense interpolation technique for encoding partial temporal ordering to obtain a single vector representation and a linear layer to obtain logits from the single vector representation. The system may use a type dependent final prediction layer.