System and method for self-supervised video transformer

Abstract

A system, computer readable medium and method trains a video transformer, using a machine learning engine, for human action recognition in a video. The method includes sampling video clips with varying temporal resolutions in global views and sampling the video clips from different spatiotemporal windows in local views. The machine learning engine is configured to match the global and local views in a framework of student-teacher networks to learn cross-view correspondence between local and global views, and to learn motion correspondence between varying temporal resolutions. The video transformer can output for display video clips in a manner that emphasizes attention to the recognized human action.