Vertex Capital LLC

Juan Carlos Riveiro Insua

David Varas Gonzalez

Delia Fernandez Cañellas

Asier Aduriz Berasategi

Elisenda Bou Balust

Joan Espadaler Rodes

An automatic video tagging system which learns from videos, their web context and comments shared on social networks is described. Massive multimedia collections are analyzed by Internet crawling and a knowledge base is maintained that updates in real time with no need of human supervision. As a result, each video is indexed with a rich set of labels and linked with other related contents. Practical applications of video recognition require a label scheme that is appealing to the end-user (i.e. obtained from social curation) and a training dataset that can be updated in real-time to be able to recognize new actions, scenes and people. To create this dataset that evolves in real-time and uses labels that are relevant to the users, a weakly-supervised deep learning approach is utilized combining both a machine-learning pre-processing stage together with a set of keywords obtained from the internet. The resulting tags combined with videos and summaries of videos are used with deep learning to train a neural network in an unsupervised manner that allows the tagging system to go from an image to a set of tags for the image and then to the visual representation of a tag.

Growing community of inventors

Menlo Park, CA, United States of America

Juan Carlos Riveiro Insua