Object-based prediction of scene transitions using neural networks

Abstract

Methods, computer systems, and apparatus, including computer programs encoded on computer storage media, for predicting scene transitions. A computer system receives an input sequence of images of a scene with each image corresponding to a different time point in an observation time sequence. For each time point, the system processes the corresponding image using a decomposition neural network to generate one or more feature representations. The system processes the feature representations for the time points using an alignment neural network to generate a set of aligned sequences of feature representations. The system further processes the set of aligned sequences of feature representations using a transition neural network to predict, for each of the aligned sequences of feature representations, one or more feature representations that represent predicted features of the object represented by the aligned sequence at one or more successive time points.